Freestyle integration into Blender

August 28, 2011

Development updates on August 28

Filed under: Update — The dev team @ 10:19 PM

Since the last blog post, several functionality improvements were made in the Freestyle branch as reported below.

Stroke geometry modifiers

A new set of line style options for modifying stroke geometry was introduced in the Parameter Editor mode.  Stroke geometry refers to the backbone polyline of a stroke.  Now stroke geometry can be modified by a list of stroke geometry modifiers specified in the Geometry tab (previously referred to as Distort).  At the moment the following geometry modifiers are available, each with a set of animateable parameters:

  • Sampling: Changes the resolution of stroke backbone polylines.
  • Bezier Curve: Replace stroke backbone with a Bezier approximation of the stroke backbone.
  • Sinus Displacement: Add sinus displacement to stroke backbone.
  • Spatial Noise: Add spatial noise to stroke backbone.
  • Perlin Noise 1D: Add one-dimensional Perlin noise to stroke backbone.
  • Perlin Noise 2D: Add two-dimensional Perlin noise to stroke backbone.
  • Backbone Stretcher: Stretch the beginning and the end of stroke backbone.
  • Tip Remover: Remove a piece of stroke at the beginning and the end of stroke backbone.

Note to branch users: When you have a .blend file with Freestyle options specified, you need to add a Sampling modifier with the Sampling parameter value set to 5, which is the default polyline resolution previously used for line drawing in Freestyle.  Before geometry modifiers were introduced, this initial sampling was automatically done.  Now the initial sampling is a tunable parameter that can be omitted, allowing better control on polyline resolution.  If no sampling modifier is specified, your line drawing will result in coarse polylines.

Images below are a few example renders made with the following stroke geometry modifiers used: (a) no geometry modifiers, showing baseline stroke backbones; (b) Bezier Curve modifier; (c) Sinus Displacement modifier; (d) Backbone Stretcher and Spatial Noise modifiers.

 

 

Fill Range by Selection

A new push button Fill Range by Selection was added to the Distance from Camera and Distance from Object modifiers (see figures below).  This button allows you to automatically fill the Range Min and Max entries based on selected mesh objects.  For the Distance from Camera modifier, the Range Min (Max) entry is filled with the minimum (maximum) distance between the active camera and the vertices of selected mesh objects.  For the Distance from Object modifier, the distance between the target object and the mesh vertices is measured.

 

Stroke splitting at material boundary

A new toggle button for splitting strokes at material boundary was added to the Strokes tab.  This option divides strokes into pieces at material boundary.  The following renders illustrate the effect of the splitting option: (a) disabling the option results in a smooth (blurred) transition of stroke colors across material boundary; (b) enabling the option permits a flat color transition at material boundary.

 

Advertisements

August 1, 2011

Development updates on July 31

Filed under: Update — The dev team @ 1:07 AM

It has been a while since the last development updates.  This absence of branch activities was mainly because of time constraints the main dev (T.K.) had during the last few months.  Due to restricted time resources, commits to the branch were mostly limited to merges of changes from the trunk and bug fixes based on problem reports from branch users.  The issues addressed include: (1) a wrong use of matrix inversion in the Distance from Object modifier; (2) composite node parameter animation not working with Freestyle; (3) a memory leak in silhouette edge detection; (4) linker errors for building blenderplayer; and (5) a bug in line caps due to zero-length strokes.  The dev team appreciate the feedback from the branch users who kindly reported these issues, some with .blend files to help us reproduce the problems.

In addition, the Parameter Editor mode of the Freestyle graphical user interface was enhanced with new Material modifiers that allow you to change the line color, alpha transparency, and line thickness of strokes by object materials.  The screenshot below shows the GUI controls of the Material color modifier.

The following set of rendering results demonstrates a typical use the Material color modifier: (a) the stroke color is constant (i.e., the base color is black); (b) the Material color modifier is applied and the base color is replaced by the the diffuse color of object materials (i.e., blend = Mix, influence = 1.0); (c) same as (b) but the diffuse color of object materials is blended into the base color (blend = Add and influence = 0.4).

   

Artwork with Freestyle for Blender

In the meantime, several branch users released excellent animation shorts and movie clips made partly with Freestyle.  First, Chris Burton and his colleagues released a finished animation short “The Light at the End“.  This is one of the most impressive pieces of artwork that give you a real sense of how line drawing with Freestyle looks like.

Dolf Veenvliet (macouno) has been using Freestyle for his Entoforms project.  Entoforms are little creatures generated by a program that “grows” them through a kind of DNA strings.  The use of Freestyle is meant to give a “drawn” look to 3D renders according to one of project blog posts.

And last but not least, Ike AhLoe released a finished animation short “Clocked In“.  During the production of this artwork, he contributed to the Freestyle integration project through a number of bug reports and development updates.

We are very grateful for the authors of these art products, since the early adoption of the Freestyle renderer should have been tough and time-consuming due to bugs and limitations.  We hope these frontier examples encourage further use of Freestyle in future 3D art projects.

March 18, 2011

Development updates on March 17

Filed under: Update — The dev team @ 2:48 AM

View map construction is the most time-consuming part of the Freestyle rendering process.  In fact, the view map construction was impossibly slow when it came to large scenes.  This was a known issue that was meant to be addressed after all to-do items were finished and the long-awaited merge into the trunk was done, mainly because of the lack of development resources.  In last December, however, the Freestyle integration project received a big code contribution from Alexander Beels.  His focus was exactly on optimizing the view map construction code.  He made this possible by identifying the performance bottlenecks and carefully redesigning the internal data processing.  Starting from the initial version in December, Alex’s optimization patch underwent several rounds of code review and testing in collaboration with the dev team.  He made substantial efforts on examining different optimization strategies and implementations.  After more than three months of dedicated research and development, his work resulted in a highly efficient view map construction code that is now available in revision 35525 of the Freestyle branch.  As illustrated with several test scenes below, the optimized view map code offers surprising performance gains.  Many thanks to Alex for the excellent achievement!

Technical overview of the optimization

Now let us describe Alex’s optimization work from a technical perspective.  The performance improvements have been mainly done in two major components of the view map construction: silhouette edge detection, and edge visibility calculation.

Silhouette edge detection is to find feature edges of interest that constitute a basis of line drawing in Freestyle.  To optimize this data processing for speed, Alex made careful changes to the code base so as to exploit compilers’ optimization capability.  This modification leads to a shorter rendering time independently of the spatial structure of a given scene.

The detected feature edges then undergo the edge visibility calculation where an integer value expressing quantitative visibility is assigned to each feature edge.  The edge visibility is determined by a raycasting algorithm with the help of a spatial grid structure that allows efficient traversal of faces in the 3D space (i.e., tris and quads) from the view point of a camera.  As a matter of fact, the non-optimized old view map code relied on a grid data structure that was implemented with the assumption that the faces were evenly distributed within the 3D bounding box.  In addition, the old code was internally creating a list of occluders by copying polygons again and again, which slowed down the view map construction.  Alex devised new grid data structures (SphericalGrid for perspective cameras and BoxGrid for orthographic cameras; automatically switched based on the camera types) that do not assume the ideal even distribution of polygons, as well as a heuristic grid density calculation algorithm that enables an adaptive population of the grid structures with polygons.  Moreover, an iterator is employed for obtaining the occluders on demand instead of making a local copy of them.

Based on these internal improvements, two optimized edge visibility calculation algorithms have been added: a “traditional” algorithm for emulating old visibility algorithms, and a “cumulative” algorithm for improved, more consistent line visibility.  Both algorithms exploit the new spatial grid data structures for fast raycasting.  Each optimized algorithm comes with two variations: culled and unculled.  The culled visibility algorithms exclude most of the feature edges outside the image boundary.  This will result in much faster visibility calculation at the cost of possible alterations to the results of edge chaining.  The unculled algorithms take all feature edges into account.  The latter is useful when off-image edges matter for chaining.

New rendering options

A new option “Raycasting Algorithm” has been added to the Freestyle tab to allow users to choose a raycasting algorithm.  Available choices are:

  • Normal Ray Casting
  • Fast Ray Casting
  • Very Fast Ray Casting
  • Culled Traditional Visibility Detection
  • Unculled Traditional Visibility Detection
  • Culled Cumulative Visibility Detection
  • Unculled Cumulative Visibility Detection

The first three algorithms are those available in the original Freestyle (the “normal” raycasting was used unconditionally, though).  The “fast” and “very fast” raycasting algorithms achieve a faster calculation at the cost of less visibility accuracy.  The other four are newly introduced optimized options.  The visibility accuracy of the “traditional” algorithms are the same with the “normal” algorithm, while that of the “cumulative” algorithms is supposed to be the same with or better than the “normal” case.  Performance improvements over the old algorithms depend on the scenes to be rendered.  The recommended options for branch users are the culled/unculled cumulative visibility algorithms.  These two options are intended to replace the other algorithms in the future.

Performance results

To conclude this brief introduction of Alex’s wonderful optimization work, some performance results are shown below using four test scenes.  In all cases, the non-optimized “normal” visibility algorithm in revision 35506 was compared with the culled cumulative visibility algorithm.

Ichiotsu (orthographic)

73448 vertices, 72260 faces, orthographic camera.

  Normal Culled cumulative
Time [sec] Time [sec] Speedup
Silhouette edge detection 5.062 3.993 1.268
View map building 14.27 1.126 12.67
Sum 19.33 5.119 3.777

Ichiotsu (perspective)

73448 vertices, 72260 faces, perspective camera.

  Normal Culled cumulative
Time [sec] Time [sec] Speedup
Silhouette edge detection 5.084 4.182 1.216
View map building 15.98 1.301 12.28
Sum 21.06 5.483 3.841

Lily

64011 vertices, 60346 faces, perspective camera.

  Normal Culled cumulative
Time [sec] Time [sec] Speedup
Silhouette edge detection 17.66 0.4280 41.27
View map building 13.07 2.422 5.397
Sum 30.74 2.850 10.78

A scene by Greg Sandor (from Lighting Challenge #15: Film Noir)

109180 vertices, 189191 faces, perspective camera.

  Normal Culled cumulative
Time [sec] Time [sec] Speedup
Silhouette edge detection 0.5670 0.3080 1.841
View map building 2045.0 10.74 190.5
Sum 2045.5 11.04 185.2

The first two examples indicate that the new grid data structures for perspective and orthographic cameras have comparable performance.  The third test showed a huge performance gain in silhouette edge detection.  The last big scene resulted in an incredible speedup rate in the view map building phase.  Remarkably, these amazing performance results are solely by means of Alex’s optimization work.  Again, thank you Alex!  This is really great!

Branch users are encouraged to test the new raycasting algorithms from both performance and artistic perspectives.  Performance reports and rendering results are highly welcome, possibly through the Freestyle thread in BlenderArtists.org.

February 27, 2011

Development updates on February 26

Filed under: Update — The dev team @ 12:39 AM

During the last two months both this blog and the integration work were quiet mainly due to very limited time resources.  However, all the changes in the trunk have been merged into the Freestyle branch on a weekly basis.  In addition, a few problems that affected many branch users have been addressed as summarized below.

1. Fixes for unexpected long straight lines

First, the issue of occasional unexpected long straight lines has been partially addressed.  Branch users may encounter this issue when a number of frames for animation is rendered.  A typical situation is that a few frames (e.g., one or two out of a hundred frames) suffer from weird long straight lines that are not in the scene data.  An example of such a long straight line is shown in the image below.  This problem is mostly due to a known bug in stroke mesh generation.  The bug is that quick U-turns in stroke topology are not handled properly.  A fix for this bug is planned as part of a comprehensive improvement of stroke mesh generation described in a blog post in last September.  As a quick remedy for now, the Parameter Editor mode has been extended so as to prevent strokes from doing quick U-turns.

Branch users writing custom style modules may be interested in the way how the fix is implemented.  Take a look at AngleLargerThanBP1D() in scripts/freestyle/style_modules/parameter_editor.py.  This predicate is used with ChainPredicateIterator() to allow two consecutive edges to be chained only when the angle of the two edges is larger than a given threshold.

In the meantime, another issue was identified as a source of unexpected long straight lines.  This issue arose when stroke thickness was specified by a curve mapping of a thickness modifier in the Parameter Editor mode.  In some occasions the curve mapping resulted in extrapolation, which gave an extremely large thickness value.  Now the extrapolation option is unconditionally disabled to prevent this issue, even when users turn on the option through the GUI.

2. Consolidation of stroke drawing

Second, consolidation of stroke drawing has been made through several related bug fixes.  After the introduction of face smoothness in last November, branch users may have seen a warning message shown below in the middle of stroke drawing:

Warning: you should not be there…

This warning turned out to be due to those edges in the view map that have a two-dimensional length almost or exactly equal to zero.  When such a zero-length edge was in a chain of edges on which a stroke was drawn, a number of style module API functions such as MaterialF0D() failed.  In addition, MaterialF0D() had another bug that an error condition was ignored due to a wrong return value.  In the end, these bugs jointly caused stroke color to be left undefined.

This stroke drawing instability has been addressed by automatically removing the zero-length edges (and chains of them) before stroke drawing begins and after stroke splitting is performed (e.g., using Operators.sequentialSplit()).  The bug in MaterialF0D was also fixed.

Moreover, possible memory leaks due to errors in user-defined style modules have been fixed.  Since scripting errors during style module coding are inevitable, these fixes improve the stability of the program regardless of having temporary programming errors in user-defined predicates, chaining iterators, and shaders.

December 29, 2010

Weekly updates December 6-26

Filed under: Update — The dev team @ 1:50 AM

The last three weeks were mainly spent for bug fixes and consolidation of the Freestyle renderer.

  • A bug in stroke rendering with round/square caps was fixed.  A freeze of the program was caused by a stroke having two vertices that reside (exactly or almost) at  the same position in the 3D space.
  • The Parameter Editor mode was improved so that it accepts quote symbols within the names of scene render layers and linesets.
  • Incorrect vertex normals in the case of smooth surfaces were fixed.  Many thanks to Stéphane for his careful code review and bug report!
  • Compilation errors with CMake on Linux were fixed by a patch from Écrivain through the BA Freestyle thread.  Thanks!

In addition, the Freestyle Python API has been slightly extended by wrappers of the following Operators class methods in the C++ layer:

  • getViewEdgeFromIndex(index): Returns a ViewEdge object in the current set of ViewEdges (0 <= index < Operators.getViewEdgesSize()).
  • getChainFromIndex(index): Returns a Chain object in the current set of Chains (0 <= index < Operators.getChainsSize()).
  • getStrokeFromIndex(index): Returns a Stroke object in the current set of Strokes (0 <= index < Operators.getStrokesSize()).

When you want to iterate over all objects in the current set of ViewEdges, Chains, or Strokes (for instance, in order to find the minimum and maximum Z depths, or sum up the lengths of all strokes), these API functions give a procedural programming way of accessing the objects, instead of relying on implicit side-effects of declarative rules (such as predicates and shaders).

December 5, 2010

Weekly update November 29-December 5

Filed under: Update — The dev team @ 11:26 PM

Since the last blog update, the dev team was working on bug fixes, addition of new features, and consolidation of existing functionalities.

A compilation issue with MinGW32 was fixed thanks to a report from Dazzle, and a bug in Distance from Object modifiers due to a recent change in the trunk about matrix-vector multiplication was fixed based on a .blend file provided by Jonathan Hudson.  We really appreciate the user feedback, which helps the dev team to improve the software.

In the meantime, the Parameter Editor mode was extended with two new feature edge selection options.  One is the “Selection by Group” option and the other is the “Selection by Image Border” option, both in part of linesets in the Parameter Editor mode, as shown in the screenshot below.

The Selection by Group option allows you to select feature edges based on a group of objects (defined through the Groups tab in the Object buttons).  For each lineset, you can specify one group of objects.  The group-based feature edge selection is either inclusive or exclusive.  If inclusive, feature edges are selected when they belong to some object in the group; otherwise, those feature edges not belonging to any object in the group are selected.  When no group is specified, feature edges are selected from all objects in object layers enabled in a render layer.  In the past, render layers were the only way to apply different line styles to different objects in the same scene.  This required the use of the compositor and thus a more and less complicated workflow.  The group-based feature edge selection is intended to address this difficulty, making object-specific line stylization much easier.  The image below is a result using this new feature.  A group of objects comprising a lily in the middle was defined, based on which white lines were drawn.  A different line style was applied to lines of two lilies in the left and right using a separate object group.

The Selection by Image Border option is used to select feature edges within the border of the image being rendered.  The border is defined either by the frame size or a border region (specified by the Shift-B key in a 3D View window).  When you work on a big scene with a lot of objects outside the viewing frustum, enabling this option leads to less memory consumption and possibly shorter rendering time.

Finally, the data flow within the Freestyle renderer was revised with the aim of reducing the amount of memory consumption.  Rendering of a 3D scene in Freestyle involves several data structures holding different representations of the scene and strokes generated from the scene.  The data processing in Freestyle is summarized as follows:

  1. First of all, mesh data are imported from the 3D scene and stored in a tree-based data structure.
  2. The mesh data are transformed into a data structure called winged edges and stored in a 3D grid structure for efficient spatial search.
  3. The winged edges are used for view map calculation, and a separate view map data structure is constructed.
  4. The view map is fed to stroke generation by means of style modules (it is noted that the Parameter Editor mode is implemented as a built-in style module).
  5. Generated strokes are converted into Blender mesh data for stroke rendering.

Previously, all the data structures were kept until the end of the entire rendering process.  This makes sense in the original, stand-alone Freestyle program, in which the view map, for instance, can be reused for applying different style modules one after another.  This is not the case in Freestyle for Blender, where the view map is recalculated for each render.  Therefore, the memory resources allocated for the data structures should be released soon after they become unnecessary even in the middle of the rendering process.  For this reason, memory usage improvements were made with regard to the aforementioned data structures.  The improvements were verified by a test scene (generated using a modified version of “Arc Sphere.es” in Structure Synth) consisting of 34968 vertices and 52452 faces.  The left image below is a rendering result.  Using a 64-bit Windows machine, memory consumption in the latest revision 33485 was compared with that in revision 33434 without the memory-related updates.  The right image below shows the amount of allocated memory (in the vertical axis) throughout the rendering time (in the horizontal axis).  The maximum amount of allocated memory in the older revision was about 2.138 GB, which reduced to 1.439 GB through the memory usage optimization as well as the Selection by Image Border option.

 

Experiments with several other test scenes showed that the extent of memory usage reduction depends on scenes.  We hope the reported updates will allow branch users to render big scenes without an out-of-memory crash.

November 28, 2010

Weekly update November 15-28

Filed under: Update — The dev team @ 1:15 AM

The dev team was working on a bug fix during the last two weeks in close collaboration with Stéphane, the main developer of the original Freestyle program.  The problem was that the view map calculation did not take mesh face smoothness into account.

So far, mesh geometry data was treated as if all faces were flat (i.e., face smoothness was just ignored), which implied that the view map was composed of jagged feature edges as illustrated in the example below.  The left image is a rendering result of a torus mesh, where strokes are drawn along visible and hidden silhouette lines of the torus.  The right image shows the feature edges (highlighted) from a three-dimensional non-camera viewpoint.  These jagged polylines are what constitute the strokes that appear in the render.  As you see, the feature edges exactly lie on edges in the mesh geometry data.  This causes somehow jaggy strokes in the rendering result (especially in the part of the render where strokes make a quick turn).

 

When the face smoothness is taken into account in the view map calculation, the resulting feature edges totally differ from the outcome shown above.  As seen in the example below,  the three-dimensional feature edges representing the silhouette lines are much smoother.  The left image is a render of the same torus mesh, whereas the right image shows the smooth feature edges resulting from the view map calculated by accounting for the smoothness property of mesh faces.  Note that the smooth feature edges are no longer constrained by the original mesh topology.

 

As far as the dev team has experimented with several test scenes, taking account of mesh face smoothness results in a cleaner stroke topology.  The downside is that the visibility of smooth silhouette lines is mathematically not well-defined.  This means that rendering results may suffer from some artifact due to incorrect line visibility.  To our knowledge, the visibility of smooth silhouettes is still an open problem in computer graphics research.  Therefore, we decided to introduce an option for enabling/disabling mesh face smoothness in the view map calculation.  In the Freestyle branch revision 33350 and later, you have the new option “Face Smoothness” in both the Parameter Editor mode and Python Scripting mode.  When the option is enabled, the view map calculation accounts for face smoothness.  This option is intended to allow users to try the two ways of view map calculation with and without face smoothness and find which one would fit individual application needs.  For now the option is disabled by default in order to avoid the aforementioned visibility issue.

November 15, 2010

Weekly update October 25-November 14

Filed under: Update — The dev team @ 12:54 AM

The dev team was working on bug fixes and stability improvements during the last three weeks.

First, a fatal bug in the internal representation of curves (e.g., chained feature edges, and strokes) was fixed.  The bug was in the constructor of the CurvePoint class in the C++ layer, and in fact it was the source of several similar bugs.  For ease of future bug hunting, a fatal error in the CurvePoint constructor will result in a verbose error report in the console.  If you encounter a crash of this kind, please let us know.

Second, a bug in the SpatialNoiseShader was fixed.  The SpatialNoiseShader is used by standard style modules to add noise distortion to strokes.  The symptom of this bug was that some strokes resulting from the SpatialNoiseShader occasionally disappear from a render.  The problem was caused by a signed integer overflow in the PseudoNoise generator used by the shader.  When image resolution was very large (2000-by-2000 for instance), the buggy PseudoNoise generator could yield a series of huge noise values, which tossed the distorted strokes out from the image boundary.

Finally, a proper support for relative paths of style module files was implemented.  It is recalled that in Blender, file paths are either absolute or relative from the directory in which a .blend file resides.  When relative paths are used, it is a user’s responsibility to keep the relative paths valid.  To help you maintain valid relative paths, Blender offers you the “Remap Relative” option when you save a .blend file.  Suppose that you have created a new scene from scratch (based on the default scene after startup).  When you save the new scene for the first time, the default .blend file path will be $HOME/untitled.blend, where $HOME refers to a platform-specific home directory.  If you have relative paths in the new scene data, they are relative from $HOME.  Therefore, if you want to save the .blend file into a different directory, it is your care to update the relative paths accordingly (otherwise, the relative paths will no longer be valid).  The ”Remap Relative” option does these updates of relative paths automatically.  And from revision 33068 of the Freestyle branch, the “Remap Relative” option takes care of relative paths of style module files.  In addition, the following operations work properly with respect to style module files:

  • File >> External Data >> Make All Paths Relative
  • File >> External Data >> Make All Paths Absolute
  • File >> External Data >> Report Missing Files

October 25, 2010

Weekly update October 18-24

Filed under: Update — The dev team @ 11:57 PM

In addition to the support for line caps, a set of GUI elements for dashed line was introduced in the Strokes panel of the Freestyle Line Style tab.

By enabling the Dashed Line toggle button, you can specify three pairs of dash and gap lengths in the unit of pixels.  Dash values define the lengths of dash strokes, while gap values specify intervals between two dashes.  If a zero gap is specified, then the corresponding dash is ignored even if it has a non-zero value.  Dashes are treated as separate strokes, meaning that you can apply line caps, as well as color, alpha and thickness modifiers.  The image below shows a few examples of dashed lines on the default cube.

Panel (a) shows simple dashed lines along hidden edges.  Panel (b) shows an example of dashed lines defined by two dash-gap pairs.  Panels (c) and (d) illustrate dashes with round caps.  Panels (e) and (f) are results of Along Stroke thickness modifiers applied to dashed lines.

October 18, 2010

Weekly update September 22-October 17

Filed under: Update — The dev team @ 1:25 AM

In the last blog post, we discussed how the complexity of internal stroke representations may increase when line joins, line caps and extreme thickness are taken into account.  After that, in the BlenderArtists Freestyle thread a separate discussion took place about Python scripting for implementing line caps.  This reminded the dev team that the implementation of different line caps is not that difficult and can be done independently of the other two issues of line joins and extreme thickness.  Hence, a new set of options for controlling the line caps has been added as shown in the screenshot below:

The Strokes panel of the Freestyle Line Style tab now allows you to use three types of line caps: Butt (flat), Round (half-circle), and Square (flat and extended).  See the following example images for each type of line caps.

  

Another small addition to the Strokes panel is the Same Object option.  By default, feature edges from different objects may be connected to form a stroke.  The Same Object option alters this default chaining rule, requesting that feature edges be connected only when they belong to the same object.

« Newer PostsOlder Posts »

Blog at WordPress.com.