In the last week the dev team was working on instability issues regarding the view map creation. The work started with a thorough review of the entire view map construction process, which consists of 1) importing meshes from a 3D scene, 2) detecting feature edges, and 3) building a view map. These steps are briefly summarized as follows.
- In the first step, Freestyle receives mesh data from Blender in the internal mesh structure called vlak. In the vlak data structure, mesh vertices are in the camera coordinate system. Since Freestyle’s feature edge detection algorithm expects mesh data to be in the world coordinate system, the vertices are transformed from the camera to the world coordinate system during the mesh importing.
- In the second step, the nature of each edge of the imported meshes is examined and relevant feature edge types (such as borders, crease edges, ridges/valleys, and suggestive contours) are assigned to the edge.
- In the third step, a 2D representation of the scene in the camera view is constructed. All 3D vertices are transformed into the 2D space by means of a model-view matrix (from world to camera), a projection matrix (from camera to retina), and a viewport specification (from retina to image). An important task in this step is to detect intersections of feature edges in the 2D space. In the case of Blender’s default cube and camera, for example, we have two intersections of edges in the 2D space from the camera’s viewpoint (as shown by circles in the figure below). The intersections of feature edges are computed using a traditional sweep line algorithm. In addition, for each intersection of two edges in the 2D space, we find the corresponding 3D point on each edge by inversely applying the viewport specification, projection matrix, and model-view matrix.
We have identified two implementation issues in the 2D-to-3D conversion of edge intersections. First, this conversion obviously does not work when edges are out of the viewing frustum. One solution to address this issue is to clip the imported meshes by the viewing frustum. A drawback of this approach is that feature edges are no longer available outside the camera view. For instance, if we have feature edges outside the camera view, we can obtain a continuous stroke that goes out and comes into the camera view as illustrated below. The left image shows a 3D scene with two rectangular parallelpiped meshes and the viewing frustum. The center image shows the camera view, in which the rectangular parallelpiped meshes are partly sticking out of the viewing frustum. The right image shows a rendering result using external_contour_sketchy.py (with minor changes). This style module draws a continuous stroke around external contours several times. Since feature edges are available outside the viewing frustum, the resulting render gives us an impression that the strokes continue outside the camera view.
If we proceed with the clipping of imported meshes by the viewing frustum, we will get a different rendering result as illustrated below. The left image shows a modified version of the same 3D scene as the previous example. The meshes have been clipped by the viewing frustum. The center image shows the camera view. Note that we have edges at the border of the camera view, and there are no edges around the outside of the camera view. The left image shows a rendering result. Now strokes turn at the border of the image instead of going out of and coming back into the image.
Clearly, the clipping of meshes by the viewing frustum makes a big difference. Ideally, only a minimum amount of clipping (e.g., only excluding those vertices that come behind the camera) takes place by default, and the full clipping by the viewing frustum is performed when users prefer to do so. We look into this issue more closely in the next weeks. The second issue we have identified is a bug in the current implementation of the 2D-to-3D intersection conversion, which fails in some apparently trivial cases (even when only dealing with vertices within the viewing frustum). We address this latter issue before we move on to the clipping problem.