Shape Files: Import & Export Blender Meshes

Talk about your new mod or map here

Moderators: bert_the_turtle, jelco

brice
level2
level2
Posts: 167
Joined: Fri Feb 23, 2007 6:04 am

Postby brice » Sun May 06, 2007 5:12 am

The GoldFish wrote:Tada? Anyway I made an awful shp->off converter in VB; it does the job.

Currently, it doesn't understand what to do with up and front (mainly because *I* don't know what to do with them, I know they relate to the orientation but I don't know how to translate that into linear space) and it doesn't support strips (although as I understand it a strip A B C D E F is triangles A B C, B C D, C D E, etc)

Nor does it do OFF->SHP, but I'm not too fussed about that right now.


Cool!

Yeah, up and front are troublesome. I'm not even sure how they would map into the 4x4 transformation matrices Blender uses. I think for importing it would be fine to leave fragments in random orientations, since the user can manipulate them pretty easily once they're in. For export I think a standardized up & front is the way to go. If you model all your fragments as separate Objects, then the DEC .off exporter will output coordinates correponding to their absolute positions in the Blender scene. Or put another way, if you have several Objects and they all have their object-centers at (0,0,0) then when you export each one (separately) you can use the fragments without any up, front, pos translations in the shape file.

You might say, well why bother with multiple fragments? You may have to if you want DGs to be able to get close to it. A single-fragment building will repel DGs based on the size of it's bounding sphere -- if the building touches the ground. So if you have a thin but tall fragment, the DGs will be excluded from a very large circular area. It gets worse the further underground the building extends. But if you chop the building up into a small base plus a tall top fragment, then the DG exclusion zone will only reflect the size of the base fragment which touches the ground. Even this is not always enough. If the top fragment is big and tall, then there needs to be a gap between it and the base -- something like 20-25 units distance -- otherwise the top piece seems to interact even though it's not touching the ground. I have one shape which I had to chop into three parts -- a base, a tall top section separated by 20 units vertical, and a small gap-filler. Without breaking it up the DGs couldn't get within a mile of the base.

Yes, strips unwind exactly as you described.

Here's an ugly little perl script that does .off --> .shp fragments. Run it with the .off filename as an argument. It does very little error checking, but it does complain if you forget to convert quads to triangles before exporting. Supposedly perl 5.8 and later will magically take care of the newline differences between linux, mac and windows. So I've read. So there's a good chance this script will run on any platform. If someone actually uses it and has a problem, let me know your platform and I'll try to figure out a fix.

Code: Select all

#!/usr/bin/perl


$infile = $ARGV[0];
if (not ($infile =~ /\.off$/) ) { print "$ARGV[1]: NOT A .off FILENAME!!\n\n"; exit;}

open INFILE, "<", $infile;
      {
            local $/; #slurp mode
            $off = <INFILE>;
      } #exit local slurp mode
close INFILE;

$infile =~ s/\.off$//;
$basename = $infile;
$outfile = $infile . ".shp";

open OUTFILE, ">", $outfile;


@lines = split("\n", $off);
($nv, $nt, $nz) = split(/\s/, $lines[1]);


$frag = <<EOH;
Fragment: $basename
   ParentName: sceneroot
      up:     0.00  1.00  0.00
      front:  0.00  0.00  1.00
      pos:    0.00  0.00  0.00
   Positions: $nv
POSN
   Normals: 0
   Colours: 1
      0:  128  128  128
   Vertices: $nv
VERT
   Triangles: $nt
TRIS

EOH


foreach $i (0 .. $nv - 1) {
   $posn .= "\t\t$i:\t" . $lines[$i + 2] . "\n";
   $vert .= "\t\t$i:\t$i  0\n";
}

foreach $i (0 .. $nt - 1) {
   $tri = $lines[$i + 2 + $nv];
   $tri =~ s/^\s*(\d)\s+//;
   if ($1 ne "3") { print "NON-TRIANGLE POLYGONS!!\n\n"; exit;}
   
   $tris .= "\t\t" . $tri . "\n";
}

$posn =~ s/\n\Z//;
$vert =~ s/\n\Z//;
$tris =~ s/\n\Z//;

$frag =~ s/POSN/$posn/;
$frag =~ s/VERT/$vert/;
$frag =~ s/TRIS/$tris/;

print OUTFILE $frag;
close OUTFILE;
User avatar
The GoldFish
level5
level5
Posts: 3961
Joined: Fri Mar 01, 2002 9:01 pm
Location: Bowl / South UK
Contact:

Postby The GoldFish » Sun May 06, 2007 5:49 am

Oh yes, I've been currently turning multifragment shapes into single.offs on the assumption that in general we didn't care. Obviously as you've just pointed out, we do, and it's a simple change. I'll keep the all-in-one functionality available or something - it might be useful, but, should I just change it to spew them out as being shapename_fragmentname.off?

For example, the radardish base appears to be not rotated, but is being scaled by the up/front values, it seems like they're doing some sort of multiplcative adjustment to the coords. I don't know anything about 3D shapes so hopefully someone out there will have a clue.

Also, I hate blender. Every time I move the mouse the entire program becomes more and more blurry. What the hell?

Anyway, I've got strips mostly working (I loaded up an armour in blender and it was fine, but globalworld_middle goes crazy), although as a single shape (getting the offsets for parents etc and shifthing things arround appropriately). I need to trawl through a few shps to find some simpler strips, and account for all the different variations (multiple strips in one fragment, strips which only have one point per line, etc)

So far it seems that shp->off is quite complicated, heh
brice
level2
level2
Posts: 167
Joined: Fri Feb 23, 2007 6:04 am

Postby brice » Sun May 06, 2007 6:28 am

"blurry"???

Do you mean big shapes (like the world map) seem to get clipped? One thing you might do is go to View > View Properties and change the value for Clip End to something big.

If you really mean blurry, you may be in a funny view mode. For mesh editing the View check marks should be on Global, Orthographic, and one of Top, Side, Front.

Blender saves the entire program state with each model file, so if you get the viewport in a funky mode, it will return whenever you reopen that file. Try exiting blender and starting with a new file to see if the blurriness persists. If it does, I have no clue.

And the radar dish is a nightmare shape to understand! I eventually got mine working, but it was a lot of random trial and error. I don't think importers should be trying to reassemble all the fragments. Well, that would be great, but to be useful you would have to create a separate blender Object for each fragment, AND then place the Objects in the correct relative positions AND parent-child relationships... all of which blender seems capable of, but I don't understand the up & front vectors, nor the blender transformation matrices enough to make it work.

There's only a few shapes with moving parts. Modding those will probably always require lots of time and a text editor.
User avatar
The GoldFish
level5
level5
Posts: 3961
Joined: Fri Mar 01, 2002 9:01 pm
Location: Bowl / South UK
Contact:

Postby The GoldFish » Sun May 06, 2007 7:00 am

What do Blender's transformation matrices have to do with anything? Ive just been applying the effects of, eg, pos directly onto each coord, to shift it around. I was hoping to do the same with up/front. Doing the necesserily math to adjust each point to switch them around onto normalised axis is no problem once I know what the numbers actually mean!

Anyway uh, I thought the radar dish was pretty simple to understand, really, it has a base, it has a dish, the dish is the child of the base, it rotates around presumably its 0 point - you just get two shapes, your base and your dish, and splice the two together.

Also, by blurry, I mean this - yes, I know it's crazy. I think the program was made by people who were insane - it was being caused by my FSAA and Anisotropic filtering settings, which I'd forced on because I was playing Oni earlier (woo Oni) - regardless it shouldn't respond that way to those settings, in fact the entire ui shouldn't even be effected by them in the slightest. Anyway I've turned them off now, so it doesn't do it any more.
brice
level2
level2
Posts: 167
Joined: Fri Feb 23, 2007 6:04 am

Postby brice » Sun May 06, 2007 7:55 am

ooo, that's burry!

The matrices would be so you didn't have to do what you're gonna do. That is: change the coordinate data directly. Then the relative effects of the parent tree get screwed up. It would be fine for static shapes, but moving ones could be thrown for a loop by the coordinate shifts when you export the data again. If every fragment is a child of sceneroot, then it doesn't matter.

Radar dish: I just remember struggling with it. A lot. Proabably because I chose it first to mod. :shock: ...I tend to dive into the deep end first. I remember having a lot of trouble getting things to rotate properly and face the way I wanted them to. But like I said I didn't really know what I was doing at the time. It's been a while since I looked at radardish.shp.
User avatar
The GoldFish
level5
level5
Posts: 3961
Joined: Fri Mar 01, 2002 9:01 pm
Location: Bowl / South UK
Contact:

Postby The GoldFish » Sun May 06, 2007 4:06 pm

Yes, but then, since .off doesn't natively support offsets to the origin, there didn't seem much point in keeping them there - there should ideally be 0 text editing between import and export. Although, I could easily add in an option which says take first coord as 0.0.0 and use it's position as pos, and subtract it from all the others (requires preplanning when making your model, but might be helpful. Usually this doesn't matter though) - really it needs some test models. to devour.

Regardless of which, I was about to add in the ability to import/export fragments as seperate.off files, so the parents etc is accounted for, although you might need to reimpliment the parentage yourself.

And as you just said yourself, there are very few moving buildings :P
brice
level2
level2
Posts: 167
Joined: Fri Feb 23, 2007 6:04 am

Postby brice » Sun May 06, 2007 8:30 pm

Two tips for exporting DEC .off files from Blender, one I've mentioned already:

1. Meshes must be converted to triangles before exporting. In Edit Mode, select all vertices (or faces) then do Mesh > Faces > Convert Quads to Triangles.

2. Darwinia seems to prefer normals facing inwards on most of the stuff I've created so far. In Edit Mode, select all vertices (or faces) then do Mesh > Normals > Recalculate Inside.

I seem to remember having some simpler stuff with normals-out, but in general you'll probably have fewer re-edits if you go for normals-in first. At least I have. It may be something the DEC .off importer / exporter is doing to the triangle orders? I don't know what was the original target for that format.

-brice
User avatar
The GoldFish
level5
level5
Posts: 3961
Joined: Fri Mar 01, 2002 9:01 pm
Location: Bowl / South UK
Contact:

Postby The GoldFish » Sun May 06, 2007 9:03 pm

I'm having some issues with strips... currently if I output a multi fragment object into a single off, which uses strips, I'm getting triangles attaching to wrong vertexes for no apparent reason.

Here's a picture

I can't see where they're coming from.

All the positions are in the right places, none of the trinangles connect randomly to positions from other sets, or ones which are in the wrong place, but, it only happens when compounding multiple fragments. It makesa no sensa!

Edit - ok, it seems that blender (and possibly the off format?) doesn't like the idea of a triangle having more than one identical vertex (3 3775 2231 2231 is in globalworld for example) and replaces the duplicate with a 0. hence why a billion triangles are all linking to 0. It might be possible to get around that... watch this space

edit 2 - yep, that's the cause of it. That issue is fixed now, and triangles with duplicate positions are no longer exported.

edit 3 - will it ever end!

Image
Image

Lots of triangles seem to have vanished - I turned off the similarity culler and nothing changed. I also added in duplicates of every triangle going v1 v2 v3 and v3 v2 v1 in case I was drawing them the wrong way.

This only affects strip->triangle conversions though. Strips are such a damn pain!
User avatar
shinygerbil
level5
level5
Posts: 4667
Joined: Wed Dec 22, 2004 10:14 pm
Location: Out, finding my own food. Also, doing the shinyBonsai Manoeuvre(tm)
Contact:

Postby shinygerbil » Sun May 06, 2007 11:16 pm

oooo...pretty Sierpinski :D
Here is my signature. Make of it what you will.
Image
User avatar
The GoldFish
level5
level5
Posts: 3961
Joined: Fri Mar 01, 2002 9:01 pm
Location: Bowl / South UK
Contact:

Postby The GoldFish » Sun May 06, 2007 11:25 pm

OK so, it IS drawing half the triangles the wrong way up, and half of them the right way up, but when you draw a triangle over the top of another one it overwrites it, hence why it didn't seem to make any difference.

Brice; do you ahve any ideas how the strips in the shp files might decide a new direction? Because I'm reading the strips off ABCDEFG -> ABC BCD CDE DEF EFG and, when I reimport I'm getting this;

Image

I'm trying to recreate this pattern, but I'm not quite sure that's going on :/

edit - ok, contrary to popular believe, strips alternate (apparently o_O), ABCDEFG -> ABC DCB CDE FED EFG, at least according to the screen grab I just got...
-- The GoldFish - member of former GIT and commander in chief of GALLAHAD. You could have done something, but it's been fixed. The end. Also, play bestgameever!
brice
level2
level2
Posts: 167
Joined: Fri Feb 23, 2007 6:04 am

Postby brice » Sun May 06, 2007 11:58 pm

Nope. I dunno. Since the shape files don't explicitly use the Normals list, the engine may be doing some heuristics to determine face directions. Above I mentioned that most of my recent blender shapes have required blender to set the normals inside. But I'm sure that hasn't always been the case. Maybe I mis-remember.

You can turn on the display of normals in mesh Edit Mode by scrolling the button window left (use the scroll wheel over the lower button window), clicking the aqua Display Normals (not vnormals), and then clicking the middle of the normal length widget and typing in 2 (the longest allowed). Then you can see what blender is doing.

You might try a double import export import export cycle to see if the problem inverts every other time. The DEC exporter may be doing something to the normals. Have a look at it's code, it's pretty clean: off_export.py under the .blender folder somwhere near the blender executable.

There may be extra rules about how triangle strips are decoded. You'll notice that often there are degenerate triangles in the list where two or three vertices are repeated for a tuple. I think these manoeuvers are to preserve the winding of the triangles (and hence the normal directions). But they may also flag the decoder for special rules. You could try googling the openGL docs since triangle strips are a native format for the openGL api, which I think darwinia targets.


Also, a few thoughts about the up / front vectors. I think these may be normalized Euler angles. If they are, then you can find all the formulas online. But there are two unknowns you may have to figure out trial and error, or by enumeration.

First, the two rotations have to be applied in some order: up then front, or front then up? Second the three values specify rotations around the X, Y, Z axes, but here again the order of ops is important... and arbitrary. Some orders are more popular than others, but Chris didn't have to follow anyone's conventions.

So there are 6 ways to order the XYZ rots, and two ways to apply the up/front rots, so if you enumerate all 12 cases and import, you should be able to pick out the correct orientation. If you try this, be sure to start with a shape off axis and in some funny orientation. A non-symmetrical shape might make it easier to tell. Just jam a couple of frags together so they have L+R handedness.

Of course, the up / front numbers may not be Euler angles. They may be shorthand matrix entries. But the way nihils page on rotations talks (http://www.nihilesthetics.info/DarwiniaModding/Tutorial_Rotation.htm), and my own experience with hand edits suggest they are related to Euler angles. They seem to exhibit the classic "gimbal lock" when I've tried to do more than one direction of rotation at a time. But what angle does a "1.00" correspond to? pi, 2pi? In some conventions, two of the Euler angles have a range of 2pi, while the third is 1pi.

These tuples also seem to work like regular vectors at times. But vectors don't have gimbal lock -- they just add. And nihils noticed they can cause shear, which kinda indicates the up / front numbers are getting stuffed into a transformation matrix. Some kinds of matrices can encode rotations, translations and shears, with the shears cropping up unwanted if you don't condition the numbers correctly (i.e. hand edit). I waffle between Euler angle and matrix entries... I have no answers, sorry.


The GoldFish wrote:edit - ok, contrary to popular believe, strips alternate (apparently o_O), ABCDEFG -> ABC DCB CDE FED EFG, at least according to the screen grab I just got...

Unfortunately you have no way of knowing the path the strip is taking across the shape. The strip can connect triangles in crazy looking paths. The programs that generate strips work hard to optimize the path to minimize the number of vertices that have to be sent to the GPU. (It's actually a very difficult problem to solve.) If crazy gets you the minimum, crazy it is! There's probably other factors involved, but the whole point of using strips is to minimize the CPU-GPU traffic by removing the redundancies around neighboring triangles.

This end of the pool gets deep quick!

-brice
User avatar
The GoldFish
level5
level5
Posts: 3961
Joined: Fri Mar 01, 2002 9:01 pm
Location: Bowl / South UK
Contact:

Postby The GoldFish » Mon May 07, 2007 1:32 am

I can understand transformations like euler angles, and the importance of the multiplicative order in matrices. I don't really need all that stuff explaining as I either already know it or don't care!

I've barely been touching blender at all, I've been working with the conversion from the shp format Darwinia uses to the .off format you've described, only occasionally loading stuff up in blender to check it's working. My work has been mainly focussed on getting any and all files from Darwinia to eg Blender and any and all models from eg Blender to Darwinia (rather than faff about with 3DS max files as well). I don't suppose you know a simple format like off which also supports RGB colour?

Anyway, openGL spec does indeed dictate that each triangle alternates in a triangle strip. Woulda been useful to know that earlier but hey, I got there in the end. I can completely understand why they use strips; do you think there will be any major performance issues with using only triangle definitions (as .offs will supply) - I don't think there will be but I'm not certain. Do you follow what I'm saying when I say ABC DCB? I'm not worried about the path the strip takes, I just needed to know that I had to reorder every other triangle definition when decoding a strip, specifically before I culled triangles with duplicate points (which aren't supported in .offs it seems).

Otherwise, I might neaten up the app (it's ugly right now), I really wanted to make it drag and drop but never really got around to it. Besides implimenting up and front transformations, as well as a colour picker possibly and selecting multiple off files to place into a single shp, it's pretty much there now - I can make an off out of any shape in Darwinia without much of an issue.

Do you know how universally .offs are supported? IE, can many other 3D editors input/output that format?
brice
level2
level2
Posts: 167
Joined: Fri Feb 23, 2007 6:04 am

Postby brice » Mon May 07, 2007 2:17 am

The GoldFish wrote:I can understand transformations like euler angles, and the importance of the multiplicative order in matrices. I don't really need all that stuff explaining as I either already know it or don't care!

Sorry, but you'll have to bear with my style. I'm verbose. Besides, there are other people following this thread, so I'm not writing just for you. And I don't know what you know.

I've barely been touching blender at all, I've been working with the conversion from the shp format Darwinia uses to the .off format you've described, only occasionally loading stuff up in blender to check it's working. My work has been mainly focussed on getting any and all files from Darwinia to eg Blender and any and all models from eg Blender to Darwinia (rather than faff about with 3DS max files as well). I don't suppose you know a simple format like off which also supports RGB colour?

The SoftImage .xsi is accesable, structured, and rich. It does have vertex colo(u)rs, but it is only export. There were a few other blender import / export pairs that used ascii formats. I just dumped out the starter cube for all of them, but I've forgotten which is which now. The trick, of course, would be getting the right triangles to end on the right vertices to pick up the vertex colors. I took a quick look at what .xsi outputs and it is much more complex that the 1:1 map that .shp uses. I think the Vertices list was Chris' way of ensuring colours got to the intended triangles -- otherwise that list makes no sense. I don't know if blender can be forced to use flat shading in a useful way. I doubt anyone asks for that feature.

Anyway, openGL spec does indeed dictate that each triangle alternates in a triangle strip. Woulda been useful to know that earlier but hey, I got there in the end. I can completely understand why they use strips; do you think there will be any major performance issues with using only triangle definitions (as .offs will supply) - I don't think there will be but I'm not certain. Do you follow what I'm saying when I say ABC DCB? I'm not worried about the path the strip takes, I just needed to know that I had to reorder every other triangle definition when decoding a strip, specifically before I culled triangles with duplicate points (which aren't supported in .offs it seems).

Gotcha. I'm curious why they alternate -- must be some performance trick or aids the stripping algorithms on average. No I don't think you'll be able to measure any performance difference at all. Darwinia is so low poly, the landscape swamps all the shapes even with hundreds of units on screen. Usually perfomance hits seem to be related to AI on screen (CPU) or pixel / glow effects (GPU?). I have a mid-range Nvidia 6600, so I tend to believe most of all performance problems are CPU side for Darwinia. (EDIT: I mean, you can set up dozens of triffids pumping out thousands of souls with little to no performance issues. But add a few DGs or engineers to the map and suddenly that many souls causes chugging. It's the AI looking for unit interactions that can't handle the souls, not the GPU.) I did notice a performance hit when I switched a spider from the default low poly shape to a triangle list with ~500 verts and ~1000 triangles -- but only when there were 100+ spiders on screen.

Otherwise, I might neaten up the app (it's ugly right now), I really wanted to make it drag and drop but never really got around to it. Besides implimenting up and front transformations, as well as a colour picker possibly and selecting multiple off files to place into a single shp, it's pretty much there now - I can make an off out of any shape in Darwinia without much of an issue.

Do you know how universally .offs are supported? IE, can many other 3D editors input/output that format?

Nope, I don't know. The header in the scripts states it is an ancient fromat from Digital Equipment Corp. A google for "off object-file-format" pulls up the date 1986, but adding "import export" doesn't pull up a lot of apps. Mostly people trying to get .off data into another format.


You are a very impressive, persistent goldfish!
PaladinOfKaos
level1
level1
Posts: 40
Joined: Mon May 07, 2007 4:35 pm

Postby PaladinOfKaos » Mon May 07, 2007 4:38 pm

Hi all,

I've been hovering around the forum for a while, finally registered since I'm finally more-or-less familiar with how Darwinia works. Anywho, I'm pretty familiar with both Blender and Python, so if someone can point me to a reference for the SHP format, I might be able to code both an importer and exporter, without messing around with intermediate formats.
User avatar
xander
level5
level5
Posts: 16869
Joined: Thu Oct 21, 2004 11:41 pm
Location: Highland, CA, USA
Contact:

Postby xander » Mon May 07, 2007 4:46 pm

http://www.nihilesthetics.info/Darwinia ... rwinia.htm
http://www.nihilesthetics.info/Darwinia ... eFiles.htm

TGF: I tried the shapes you sent me. I had forgotten how bad the problem was on my work computer. They didn't help a bit. :(

xander

Return to “Mod Projects”

Who is online

Users browsing this forum: No registered users and 2 guests