Shape Files: Import & Export Blender Meshes

[PaladinOfKaos]

I'd like to keep the initial importer talk here, so the other thread doesn't get too big too fast. Once the importer is stable, we can move it there as well.

[brice]

* Object names are used to name fragments. I'm not certain if blender's 'name.001', 'name.002' duplication scheme will work with Darwinia. I Still haven't tested it.

I does work fine. At least in the few simple tests I've done.

In most cases, Darwinia doesn't care if fragments (or Markers) have unique names. You can add 20 ports by duplicating "MarkerPort01" 20 times without changing the serial number. You don't even need trailing serial numbers. And you can have multiple fragments with the same names, and all seem to get rendered ...most of the time. I think the exact names of some fragments and the exact Marker serial numbers sometimes matter. But in general, the Blender naming for duplicate should work just fine.

[brice]

Here are a couple of pretty pictures showing multiple Blender Objects getting exported to multiple fragments within a single shape file. The Blender parent hierarchy is properly exported as a darwinia ParentName hierarchy, with all transformations intact.

The circled blender object is the root parent and is exported with ParentName sceneroot. The dotted lines indicate child relationships. Each one applies rotations and translations to it's children. The deepest child is shown in Face Paint Mode so you can better match it up with the Darwinia render.

And as a bonus, every blender Object is what's called a Linked Duplicate of the root parent Object. This means they all share common mesh data, so edits to the common mesh are reflected at every copy. They're effectively dynamic clones. The exporter handles this transparently.






PaladinOfKaos: can you add some blank lines between fragments? Also, when you export Markers, don't forget to count the child Depth. In some shapes it doesn't seem neccessary, but in others, like radardish.shp it is crucial. Depth = tree depth with the root node starting at 1 (or equivalently, sceneroot = 0). So if a Marker is a child of the last child Object shown above, it would have Depth: 6.

[The GoldFish]

My advice re the importer, split it into 2 seperate projects - an importer script which doesn't believe in strips and only loads in triangle definitions, then make a 2nd script which converts strips to triangle definitions.

And then combine them together at some stage... or something.

I'll point out again that (as I recall) many of the Darwinia shapes are available somewhere as 3DS files, so are oftentimes importable anyway. It's not ideal though.

[brice]

On the IV Darwinia pages http://www.darwinia.co.uk/community/modding.html you can get a zip of the original .max files. Unfortunately that 3DS format is proprietary and no one can implement an importer for it without a license. I scoured google for converters, and that's the consensus I found.

Blender can import .3ds files, but no one has a .max -> .3ds converter without owning a copy of MAX. At least that's the way I understand it.

If anyone has a zip of converted .3ds shape files, it would be great to try to import them with the current Blender .3ds importer. That could save PaladinOfKaos some work.

I guess there's not much point to a .shp importer once we get all the existing shapes into an importable format, or into .blend files. Then anyone can use any of Blender's exporters if they want to use another 3D modelling program.

But on the other hand, there are several new shapes made by modders. If someone wants to import those into Blender, then a .shp importer is the only way to go... unless you want to hand-edit a .off file of the coordinates and triangles.

[The GoldFish]

We already have a basic shp->off converter, or did you forget already!

A program or script which simply converts strips to triangles is also easily within grasp. Just a basic reverse version of PaladinOfKaos's export script would be perfect, having strip support in that script is just a small bonus.

Also sorry, I didn't realise they were only available in max. I know people with 3DS who should be able to export them, but I don't know that they care.

[brice]

We already have a basic shp->off converter, or did you forget already!

Ooops! Well, I'm on linux, and I know I can probably find the stuff to run VB scripts (wine, etc.), but it's probably less work for me to hand edit the .off files I need. But I know I'm in a small minority on this. Didn't mean to slight you!

If we can get an importable library of existing shapes, then the need for a .shp importer goes away. But PaladinOfKaos might feel differently?

[brice]

PaladinOfKaos: your exporter is not applying Object scale factors. So a mesh is exported with it's raw coordinate size, as if the scale were always 1.0 in the Object data.

Also... the exporter can be quite slow when dealing with a few thousand verts and faces. I'm on a 1.8GHz Athlon and it can take 5-15 sec to finish.

There is a WaitCursor function wich should probably be called:

Code: Select all

Blender.Window.WaitCursor(1)
    do stuff
Blender.Window.WaitCursor(0)
Blender.Window.RedrawAll()

And there is info here http://en.wikibooks.org/wiki/Blender_3D:_Blending_Into_Python/Optimize about optimal API use for exporters. This in particular might be useful:

Code: Select all

Use new python functionality mesh.transform() (Exporting Only)

As of Blender 2.37 nmesh.transform(ob.matrix) can be used to transform an NMesh by a 4x4 transformation matrix.
On my system it is over 100 times faster than transforming each vert's location and normal by a matrix within python. Though the overall speed gained by using this in my obj exporter was about %5.

Like the intro says, don't worry about optimizing till it's all working. But it might be helpful to skim through this page to see where the pitfalls are. I'm assuming you're doing all this work -- in part -- as a learning experience.

[PaladinOfKaos]

I should be able to apply scale factors pretty easily, but that's another multiplication in what, as you say, is already a slow script. I'll definately add the WaitCursor() calls, though. I may be able to add other optimisations - perhaps keeping more stuff in the string before the write() calls.

I should probably also use Psyco, that'll make it quite a bit faster. That'll also be the first external dependancy, which is something I'd hoped to avoid.

TGF: Are you using VB6 or VB.NET? If it's .NET, it should only take a bit of tweaking to make it run under Mono on Linux.

[brice]

I should probably also use Psyco, that'll make it quite a bit faster. That'll also be the first external dependancy, which is something I'd hoped to avoid.

Yeah, let's avoid dependencies for now. Most Darwinia shapes will be fairly low-poly, in all likelyhood. As long as there is a wait cursor, and some mention of the slowness on the DL thread, I don't see any problems. I just mentioned it so you'd know. If I or someone else wants to do higher poly stuff, 10sec export is a small price to pay for one-click conversion to .shp! (Oh damn! is One-Click still patented?)

[brice]

One more minor issue. Your script is culling duplicate points and degenerate faces, right? It seems to be. This is good in most cases, except one which is useful for shape modding.

The height of the glow / mist effects is usually set by the median height of all the Positions of all the fragments (or something along these lines). This works for SpawnPoint, ConstructionYard, TrunkPort, Goddish. If you want to move the glow effect up, you must include a dummy point to displace the median. Sometimes this can be done with an empty fragment whose pos: displaces the overall shape median, but sometimes it is better to include a dummy point within a fragment itself. The difference is meaningful when trying to place ports near a building. Ports are displaced automatically to avoid the bounding spheres of fragments, and a high empty fragment can cause unwanted port spreads which an identically placed dummy point does not. (The exact rules are more than a little opaque.)

So if I try to include a dummy point in my blender model, your exporter culls it because it is part of no faces. If instead I include a degernerate triangle with three identical vertices, your exporter culls 2/3 verts, but exports the triangle using the remaining point. However if I remove the duplicates in blender, your exporter no longer exports the remaining point or the triangle. So I have to remember to not remove the dups for those three points way off in space.

This is not a big deal, but I wanted to point it out. Maybe you can relax the culling slightly? Blender already has a remove duplicate points function that users can apply manually. Why apply the same function during export? Not a gripe, just a thought.


P.S. I PM'd Icepick to see if he can sticky your DL thread. I don't think mods or admins come here often. None are listed on the front page for this forum.

[brice]

PaladinOfKaos: your script continues to work great! But here's a bugfix and a very minor addition.

To add the needed colon, change line 153 to:

Code: Select all

    file.write('\tParentName: '+self.ParentName+'\n')

And to add whitespace between frags and markers change this region of mainFunc:

Code: Select all

  for frag in shp_fragments:
    frag.write(file)
    file.write('\n')
  file.write('\n')
  for mrk in shp_markers:
    mrk.write(file)
    file.write('\n')
  file.write('\n')
  file.close()

Also your .py header lists the wrong version number, viz the DL page.

Also, a thought about the naming of Markers and using meshes to represent them. One marker in radardish.shp has a very long name: MarkerTeleportEntrance. It's too long for Blender's name field. I don't know if this is a unique situation. But if you decide to use Linked Duplicates of a common marker mesh, couldn't you trace that linkage to figure out who are the Markers? Then we could leave the initial "Marker" out of the name field and add it in only on export.

EDIT: Any linked clone of the generic Marker Mesh Object would be considered a Marker. The generic mesh could be a marker itself, or just a placeholder without real meaning. Would there be a way to lock the generic so it doesn't get used? Or you could simply compare each selected Object mesh to the known form of the Marker Mesh, assuming you choose a simple wireframe or camera-box like thing. Anything that looks like a Marker *is* a Marker. Who really needs to customize their marker mesh?

Thanks again for the script!

EDIT: also, FYI, there is no problem writing out parented fragments in arbitrary order. The .shp parser doesn't seem to care about the tree order -- probably just building a dictionary.

[brice]

PaladinOfKaos: if you ever come back, here's a python triangle stripper you can probably hack into your script. It comes from the TorqueDTS exporter for Blender: http://projects.blender.org/projects/torqueexporter/ The code has a very liberal license, so you can probably just cut and paste into your own exporter:

Code: Select all

QADTriStripper.py
Quick and Dirty Triangle Stripper, uses a simple greedy heuristic.

Copyright (c) 2006 - 2007 Joseph Greenawalt

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

The TorqueDTS exporter also interfaces to the VTK library's stripper. I'm playing with that now. Unfortunately my python bindings for VTK are python 2.3 and the Blender exporters are bound to 2.4 on my system... and I'm unwilling to resolve the dependencies to make the two pythons one. In any case I'm much more comfortable using perl to hack intermediate file formats.

I've learned several things about triangle strips so far. The VTK stripper outputs lots of short triangle strips. A close re-reading of http://www.codercorner.com/Strips.htm, especially the last two sections "Backface Culling" & "Connecting Strips" shows how to merge all these short strips into one long strip. Long strips give the best efficiency / performance gains. My own experimenting has boiled this down to a couple of simple heuristics, which may be useful if the above QAD stripper also generates short strips:

1. Double up the first and last strip vertices for each short strip, except don't double the first vertex of the first strip (or the last vertex of the last strip, but this is less important). When concatenated, these extra vertices will form the "void" pairs needed to join the strips.
2. Locate the strips with odd vertex counts. Insert ONE extra "void" vertex AFTER each odd length strip. This can be a doubling of the last vertex of the odd length strip, or a doubling of the first vertex of the next strip. Don't bother reversing any strips. Don't worry about doubled and tripled vertices.
3. Concatenate the vertex lists and get a new count for the shape file strip header.

That's it! The joined vertex list will preserve the winding orientations of each sub-strip, so all normals will point out (if they originally did). The extra doubled and tripled vertices will cause no performance degredation since this is a common trick and the OpenGL rendering pipeline culls degenerate triangles early on. This trick may be the cause of all / most of the doubles & triples found in the original Darwinia triangle strip shapes.

I am currently putting together a crude perl script which will do the following:

1. read in .shp
2. output .vtk intermediate
3. call short python-VTK script
4. read in .vtk triangle strips
5. merge strips
6. output new .shp

I will post it (when it works) in case anyone else can use it. It will require you to have both python, VTK and python-VTK bindings installed on your system. For linux boxes, installing the python-vtk package will take care of all your dependencies (but might use a python version different from what your blender uses). For Windows, go to http://www.vtk.org/index.php to download a VTK installation, and to search for python binding info. I think the VTK installation includes the bindings, but your python would have to match versions. Or something.

Anyhow, real triangle strips don't seem that far off for the blender exporter.

But for low-poly shape fragments there is no need to go to any of this trouble. I was trying to build a render-efficient level-3 Menger sponge shape. With some time-consuming blender work I was able to reduce the triangle count 30% from my first naive approach. With a stripified shape, the total vert count for rendering should go down by another large fraction. Otherwise a level-3 sponge is too complex for Darwinia -- probably the hidden face culling or something. It renders fine alone, but when other stuff is nearby, rendering stutters, even though the fps doesn't go down appreciably. The shape file itself is no bigger than the world maps, but they don't have fractal nesting of faces.

EDIT:

...and here is the perl script. It is crude and does very little error checking, but it works. I have successfully converted almost every shape file I have collected to triangle strips. There are a handful of shape fragments which the VTK stripper chokes on for mysterious reasons -- probably a bug and I've logged a bug report -- but the perl script simply retains the original triangle lists when the stripper fails. Performance for my level-3 Menger sponge is noticeably improved, although rendering still stutters faintly when the camera pans around that building. I'm sure it's a pathological worst case for the rendering pipeline.

Usage for the script is as follows: "perl stripify.pl filelist" where file list can be wildcarded like "*.shp" to do batch conversions. All filenames are expected to end in ".shp", but this limitation can be easily fixed. The script tries to convert every fragment of every file unless it already uses triangle strips. Normally the converted file is output to "filename.shp.strip", but if you comment out the line indicated near the end of the script, it will replace the .shp files as it goes. Be sure to have a backup copy of all your shapes before you muck around with this script. Please!

Code: Select all

#!/usr/bin/perl

#first dump out the python-vtk script so we know it's there
$python = <<EOPY;
from vtk import vtkPolyDataReader, vtkStripper, vtkPolyDataWriter

reader = vtkPolyDataReader()
reader.SetFileName("stripify-vtkinput.vtk")

stripper = vtkStripper()
stripper.SetInput(reader.GetOutput())
stripper.Update()

newmesh = None
newmesh = stripper.GetOutput()

writer = vtkPolyDataWriter()
writer.SetInput(newmesh)
writer.SetFileName("stripify-vtkoutput.vtk")
writer.Write()
EOPY

open PYFILE, ">", "stripify.py";
print PYFILE $python;
close PYFILE;


#loop over all files listed on command line (including *.shp globs)

while (@ARGV) {
   
   $file = shift @ARGV;       
   open INFILE, "<", $file;
   {   local $/; #slurp mode
      $shape = <INFILE>;
   } #exit local slurp mode
   close INFILE;
   
   $reshp = qr/
                  
         Positions \s*? : \s*? (\d+)
         (.*?)
         Normals
         
         .*?
         
         (^ \s* Triangles \s*? : .*?)
         
         (?: Fragment | Marker | \Z )
            
         /imosx;
   
   $shapecopy = $shape; #since we're substituting back into it on the fly
   while (scalar $shapecopy =~ /$reshp/g ) {
      
      #use regex to extract each successive fragment's poly data   
   
      $poscount = $1;
      $poslist = $2;
      $triblock = $3;
      
      $triblock =~ /Triangles \s*? : \s*? (\d+) (.*)/imosx;
      $tricount = $1;
      $trilist = $2;
   
      $poslist =~ s/ ^\s* \d+ \s*: //gmx;    #remove pos numbers
      $poslist =~ s/ \, / /gx;            #commas --> spaces
      $poslist =~ s/ \# .*? $ //gmx;          #remove comments
      
      $trilist =~ s/ \, / /gx;            #commas --> spaces
      $trilist =~ s/ \# .*? $ //gmx;          #remove comments
   
      # remove degenerate triangles -- some original darwinia shapes have them
      
      $goodtris = "";
      $tricount = 0; #recount
      while (scalar $trilist =~ /^\s* (\d+) \s+ (\d+) \s+ (\d+) \s*$/gmsox ) {
         ($a, $b, $c) = ($1, $2, $3);
         if ($a != $b and $a != $c and $b != $c) {
            $goodtris .= "3 $a $b $c\n";
            $tricount++;
         }
      }
      
      $trilist = $goodtris;
      $numints = 4 * $tricount;
            
      #substitute everything into a minimal .vtk file format
      
$vtk = <<EOT;
# vtk DataFile Version 3.0
vtk PolyData input = shape Positions + Triangles
ASCII
DATASET POLYDATA
POINTS $poscount float
$poslist
POLYGONS $tricount $numints
$trilist

EOT

      $vtk =~ s/ ^ \s* //mgx; #remove any blank lines

      #save .vtk temp file
      
      open VTKFILE, ">", "stripify-vtkinput.vtk";
      print VTKFILE $vtk;
      close VTKFILE;
      
      open VTKFILE, ">", "stripify-vtkoutput.vtk";
      print VTKFILE ""; #emoty file so can detect vtk failures
      close VTKFILE;
      
      #call out to python-vtk to generate the triangle strips

      print "Pythyon-VTK call # ", ++$call, ": $file...", "<"x40, "\n";      
      
      system ("python stripify.py");
      
      #retrieve .vtk strip file
      
      open VTKFILE, "<", "stripify-vtkoutput.vtk";
      {   local $/; #slurp mode
         $vtkstrips = <VTKFILE>;
      } #exit local slurp mode
      close VTKFILE;
      
      if ($vtkstrips eq "") { #empty result = silent stripper failure
         print "!!Stripper Failed on a fragment in $file : Triangle list retained.\n";
      } else {
         
         #build Strips: block, joining all sub-strips
         
         $revtk = qr/
                  TRIANGLE_STRIPS .*? $ #eat entire line
                  
                  (.*?)
                  
                  ^ \s? $ #blank line
                  
                  /imosx;
         
         $vtkstrips =~ /$revtk/;
         $strips = $1;
         $strips =~ s/ ^ \s* //mgx; #remove any blank lines
         @strips = split("\n", $strips);
         
         @longstrip = (); $i = 0;
         foreach $strip (@strips) {
            
            @verts = $strip =~ /(\d+)/g;
            $vcount = shift @verts;
         
            if ($i != 0) { push @longstrip, $verts[0] } #double first vert
            push @longstrip, @verts;
            push @longstrip, $verts[-1]; #double last vert
            
            if ($vcount % 2) { push @longstrip, $verts[-1] } #invert winding if odd
         
            $i++
         }
         
         $numverts = $#longstrip + 1;
         $vertlist = join(" ", @longstrip);
         $vertlist =~ s/ (\d+) / v\1 /gx; #prefix the v's
         
         $vertlist =~ s/ (.{1,80} \s) /\1\n\t\t/gx; #insert line breaks
         $vertlist = "\t\t" . $vertlist;
         
         #substitute Strips: for Triangles: block

$stripblock = <<EOS;
   Strips: 1
      Strip: 0
         Material: GunMetal
         Verts: $numverts
$vertlist
EOS

         #print "$file: \n", $stripblock, "\n\n";
         $shape =~ s/\Q$triblock/$stripblock/; #\Q protects against embedded comments
   
      }#if non-empty
         
   }#while fragments

   #write out the new stripified version of the file   

   $outfile = $file;
   
   # comment out the following line to
   # replace .shp files instead of
   # creating .shp.strip files
   ##################################
   $outfile =~ s/\.shp$/.shp.strip/i;
   ##################################
   
   open OUTFILE, ">", $outfile;
   print OUTFILE $shape, "\n\n";
   close OUTFILE;
   
}#while files


[PaladinOfKaos]

Thanks for the info, Brice. Real life's been a real pain lately, hopefully I'll have time to work on the scripts some more soon. Only a couple more weeks of school, but hopefully I'll have free time again before that.

[Maggy]

Please forgive me for being a little off topic.
Dassault has released two very nice freeware utilities, a 3D printscreen that captures both openGL and directX shapes and textures from almost any 3D software and a 3D viewer for the files created by the 3D printscreen, which are in 3DXML format.
http://www.3ds.com/products-solutions/3d-for-all/

So far the good news, the bad news is that there are so far no converters from 3DXML to any editable format. Even Dassaults flagship Catia that produces 3DXML files refuses to open the 3D printscreen files for editing.
Good news: the printscreen files are ordinary zip files. The 3DXML within these zips are for the major part very readable XML, Fans, strips, triangles, faces, vertex buffers, normals and so on in plain ASCII.
Bad news: I'm not a programmer
Good news: someone already sent me a tiny python script that finds triangles and converts them to .obj files that I can read.
Bad news: just plain triangles, no strips, no fans and so on. It does convert my Sketchup screenshots, it doesn't do most other screenshots.

More bad news: All I want is a way to capture shapes and import them into Sketchup. I seriously tried to dive deep into that python script, the 3DXML files and the theory behind triangle strips and so on, tried to find a way to rewrite the script and failed miserably.

The only good news left is that what I've read in this thread in this forum makes me believe that there are some serious experts here on this very subject who should in theory be able to write a 3DXML to whatever converter. To whatever because I do not need specifically .obj or .dxf or .wrl or .3ds or.... as long as it's a format that I can convert to any of these common formats.