Windows vs. Linux Performance?? Please Help...

[brice]

Hi, Does anyone have any experience running mods on BOTH linux and Windows? Preferably dual-boot on the same machine. Does Darwinia Windows always run slower than Darwinia Linux on the same machine? I've been developing a mod under linux for a few months, and I just installed Darwinia for Windows. Some of my test levels bog down much sooner under Windows, than under linux. I was not expecting this. Is it normal? Or if not linux, does Darwinia on a bootcamp Mac generally outperform XP on the same machine?

I have reviewed all the stuff on the Windows Issues forum. I updated my Nvidia drivers and sound drivers and motherboard drivers, etc. Darwinia still runs faster under linux with really old Nvidia drivers. I don't game under Windows so I don't know what else to try. Does anyone have any suggestions? I played with all the obvious Darwinia settings including Sound. I didn't try RenderMode because my card is recent enough to accept VBOs, and it's the same card under linux.

Specifically, it seems like the Windows version can't handle as many units/entities running around on the map. My system slows down and the sound starts breaking up and echoing or stuttering under the load. When the linux version boggs down the sound never breaks up like that. It doesn't seem like a memory leak, since a less complex level will run forever without slowing down. The problem is definitely related to the number of DGs, etc. that are on the map.

It seems like I can have at least twice as much stuff running under linux. This makes no sense since Darwinia is very cross platform. So I'm guessing somethings up with my driver settings or my Windows settings. Hopefully someone will be able to point me in the right direction.

I had planned to use 20fps under linux as my threshold for how much stuff I can jam into a level. But Windows is running less than 1/2 that for some roughly similar level situations. So now I don't know how to judge a good performance limit. Probably most people will have faster machines than I do, but a 50% slowdown for Windows is still a big deal. Many of my levels are complex, so I may have to redesign everything if this performance issue is real.

My specs don't really impact the problem since it's the same machine both ways, but people will want to know, so here they are. The machine is 1.8GHz Athlon, 1GB RAM, Nvidia 6600 PCIe. Nothing fancy. The linux setup runs Darwinia 1.4.0b9 with Nvidia Driver: 1.0-7174 from way back in 2005. The Windows XP setup runs fully patched Darwinia 1.42 with the latest Nvidia Version: 94.24 driver and DirectX 9.0c. Windows has been updated and updated and updated (that's what I get for not booting it in over a year!).

Thanks!
-brice

[The GoldFish]

And if you use 1.5+, your windows performance would halve again.

I think the speed order is (fastest) Linux, Windows, Mac (slowest)

I have a box next to me right now with ubuntu and XP on, I'll give Darwinia a whirl on both and give you some FPS figures. I would just run rendermark on it but I don't have a tool for linux which can record FPS to a file every second!

edit;

Ghost Capital, skipping the intro and a few seconds in, looking down at the yard towards the gladiator trunk port, on a 2.4Ghz P4 with a 5900XT

Windows 1.41 - 29fps
Ubuntu linux 1.40b9 - 60fps
Windows 1.5xbeta - 21fps

So there you go!

edit - oh, and I'm quite sure this has nothing whatsoever to do with drivers.

[shinygerbil]

I can tell you now that all 3 of Introversion's games run a lot smoother on Linux than on Windows. I always have to boot into Linux just to play Defcon!

[brice]

Windows 1.41 - 29fps
Ubuntu linux 1.40b9 - 60fps
Windows 1.5xbeta - 21fps

THANK YOU!! Thank You BOTH!! Sorry for all the exclamations, but you just saved me from wasting LOTS of time trying to tweak my Windows setup -- I was NOT looking forward to it! I already spent hours and hours enduring Updates and driver install reboots, and BSODs, etc.

Wow, I never would have expected Linux to be a gaming champ, and I thought twice about posting because I didn't want to start any religious bickering. But those numbers speak for themselves. I was guestimating 50% by when AISpawnPoint DGs would grind a particular level to a halt.

Thanks for the confirmation! I will have to redesign a few levels so the 95% Windows world can play my mod without monster machines. Probably have to break several levels apart. Good thing none of them are finished yet.

Thanks again! (did I say that already? )
-brice

[The GoldFish]

It's not really a gaming champ, it's just certain people don't know how to code effectively for a windows environment (and it's not exactly an easy thing, either)! IV's engine is, at a fundamental level, crap. All credit to them, it's far better than anything I'm going to be coming up with any time soon, but that doesn't mean it's a good engine. It really ought to be running many times faster than it does...

[brice]

I hear you. For whatever reasons, IV's engine is best matched to the linux environment. I really don't want to stoke a Windows vs. Linux controversy. I'm not religious that way. I *am* curious what linux does to make IV games so much faster -- possibly that IV codes against a lot of open source APIs so there is a reverse-porting penalty under Windows? -- but I'm in no way shouting "yea linux go linux!"

Mostly I'm glad I ran into this before I got further along with my mod -- I don't want to make it unplayable. And I am walking a thin line already in some levels. I'll have to back off a little more on the eye candy.

[brice]

... on the topic of Windows modding, is there a way to run Darwinia so you can see the server messages while it's running? Under linux, the console collects all the Darwinia messages (and errors). But when I start Darwinia from the WinXP console I get nothing -- the game detaches from the console and the command prompt returns. So when I crash with no blackbox I get literally nothing to go on.

Thanks,
-brice

[The GoldFish]

AFAIK the windows version doesn't have that, I've never needed it anyway...

The source of the decreased FPS could be to do with *anything* that's been changed between the two versions. Best bet is ask John!

Oh, and huge levels crammed with stuff isn't necesserily a good idea, anyway! Really you want a dynamic enemy force to be created around the player (which is in some ways why the 'erratic' DG behavoir where they pool and wait for you to attack is actually quite good) so the predominant enemy force is at the front line of the fight, being replenished as they fall back.

[brice]

AFAIK the windows version doesn't have that, I've never needed it anyway...

Well, try it under linux sometime. Just run Darwinia from a terminal window. You can see messages about level creation, unit instantiation, level exit / enter and mission and game file saves, and more. The mission save info helped me diagnose an infinite loop between two scripts that exited and entered two levels alternately. The messages about "EntityGrid max neighbours set to 100" correspond to slowdowns caused by DG density vs. total DG population. When that number starts going above 1000, 2000, or even 3000, the AI engine becomes jerky and camera panning will hiccup. Of course that's pretty easy to see by eye. But if you run Darwinia windowed you can watch the server messages in real time to get a feel for the timing of the problem as it builds up.

The messages I most wanted to see are the ones about which shapes caused a crash, and which buildings have UniqueId conflicts. I finally forced a UniqueId issue and realized that Windows pops up a dialog box for those. Moving my test levels from linux to Windows, I've had a lot of crashes that didn't create blackbox files. For example, under linux I can add a 12-sample soundgroup for squad lasers, while under Windows anything over 10 crashes on startup. Server messages wouldn't help diagnose that, but Darwinia is so flaky about some things that I prefer any info to none at all.

(And yes I know modded sounds require a sounds.txt replacement. But there is a way to accomplish that without un-rar-ing the main.dat... just add an empty "data" folder to the Darwinia directory and drop the new sounds.txt in there. It will override the one in main.dat without having to touch main.dat. Works on both Linux and Windows, so probably Mac too.)

If you run Darwinia from the Windows console with "darwinia >outfile.txt 2>&1" you can see messages which don't show up under linux -- they get redirected to the outfile.txt file. But they only document the progress of the *.dat file un-rar-ings.

Oh, and huge levels crammed with stuff isn't necesserily a good idea, anyway! Really you want a dynamic enemy force to be created around the player (which is in some ways why the 'erratic' DG behavoir where they pool and wait for you to attack is actually quite good) so the predominant enemy force is at the front line of the fight, being replenished as they fall back.

I totally agree with dynamism, but I disagree with the merits of the surge and retreat AItarget behaviors. I fired up your Ghost Capitol level, and the Reds never tried taking the hilltop -- all I had to do was nothing for 10min and watch the fps. (Maybe another AI change from 1.3 to 1.4?) With a lot of tweaking (and an equal amount of serendipity) I've discovered a way to get DGs to respond to player global battle strategy vs. just local tactics. Imagine you had a fortress with a tight network of AItargets. The player might approach from the NSEW. Wouldn't it be great if the DGs ran in the appropriate direction to meet the player and joined in battle *before* the player gets all the way to the center? It's possible (with several caveats).

I'm not going to describe the details here since it would spoil some of the surprises in my mod. If I mentioned some of the things I'm doing, I'm sure you could figure out how I'm doing most of them. For example, if I told you the Red Ants have stolen the raytracer from the bootloader, I'm sure you can guess which building I hacked for that role. But I think I have a few surprises for even you. If I told you I can make squaddies fly... controllably. No ego here, just don't want to spoil the surprises for people. And if it looks like I'm not going to finish the mod, I'll post a bunch of the best stuff so people can see how far the engine can be pushed, and in which directions. I'm not taunting and keeping secrets just to be a mean old bastard.

Suffice it to say that a lot of the "eye candy" slowdowns have nothing to do with DG on DG battles. I'm pushing the engine in many ways. So I'm kinda bummed that Linux Darwinia is 2x faster than Windows. It means I have to cut back a lot of neat "stuff" -- not just the size of my DG armies. Oh well, them's the breaks. I may borrow your Wus Cave idea on a few levels so people with monster machines (or linux) can see the levels the way I envisioned them. Everyone else can wus out when their fps bites.

-brice

[The GoldFish]

So you fired up ghost capital and you didn't get attacked? Sounds like whatever version of Darwinia you're running doesn't have the movement behavoir witnissed in 1.3-1.42 on windows, which I had to go out of my way to adjust the level for, because it made the red AI far far more ferocious.

Considering 1.5.x fixes the darwinia movement back to how it was before (or similar to) it probably needs fixing again...

Anyway don't expect to see the windows version speed up any time soon. Really, what I'd like to see, is *uniformity* between all the versions, but it just plain isn't going to happen.

[brice]

I'm using 1.4.0b9, just like you. Well I probably shouldn't have used that as an example. Sloppy of me. I fired up Stricken under AAA and the reds just came to the rim of the hill but no further. The green DGs kept their cannons without any help. And the squad and console were untouched by the reds. But now that I open it up in the editor I see there's no targets inside the rim, and without AAA the map is empty, so you're doing something with several missions, and I'm missing the whole experience by using AAA. I only wanted to see your fps reference scene, so I shouldn't have commented on the AI for that level without really playing it. (I did a long time ago, and enjoyed it under Darwinia 1.3.)

One of the reasons for installing the Windows version is to make sure my hacks don't depend on linux version 1.4.0b9. So far, the memory issues are slightly different under Windows -- like the crash for 10 vs. 12 sound samples, which seems related to a cap on the total number of samples allowed under each version. I'm just hoping they're stable for all machines with a given platform version. The radar dish is very flaky about adding lights to the dish itself. Add n and it crashes, add n+1 and your fine, add n+2 and it crashes again. That kind of thing must be due to memory conflicts and sloppy mallocs or similar. I'll toss out any mods that don't fly on both 1.4.0b9 and 1.42, but I'm still half expecting some people to compain that it just won't load on their machine. Oh well. Hopefully those people will be willing to revert the building shapes one by one so I know which building(s) are causing the most issues. All my hacks are in the shape files.

But there's also a lot of new stuff that can be done without hacking the buildings. Well, "new" in that no mods have used them before -- I'm sure a few people have seen some of this stuff. For instance I figured out how to detect burning trees. In fact trees are almost as useful as triffids in some cases. I always wanted to have trees do something other than stand around and look pretty. And there's lots of good uses for the two bridge buildings (although they really need new shapes to be most useful). Some things are too close to being bugs, like shadow DGs. There are two ways of making DGs walk underneath the land -- only their shadows are rendered. But both ways cause performance issues which are difficult to control for. Too bad. It's cool to watch a column of zombie shadow DGs marching along... until they take your fps down to single digits.

[brice]

UPDATE & SANITY CHECK:

Windows 1.41 - 29fps
Ubuntu linux 1.40b9 - 60fps
Windows 1.5xbeta - 21fps

... No one should take away the false impression that Linux 1.4.0b9 is 2x faster that Windows 1.42 *across the board*. It isn't. I have moved another taxing test level from Linux to Windows and it runs about the same fps both ways (maybe 2-3fps less under Windows). It uses the reciever system and has 7 SpiritProcessors, which means 7x the soul rain. It also has close to 1000 (sic) active buildings on the map (don't ask me why, but there is a real modding reason and it's very very cool!).

The massive slowdowns under Windows seem most related to large DG wars. It may be AI related, but my gut feeling at the moment is that the sound system is more likely the culprit. Like I mentioned above, under heavy stress (like 'nading a tight cluster of 1000+ DGs) the Windows DG sounds will break up and echo and get stuck in a stuttering loop (along with the fps). The same sound behavior doesn't occur under Linux, even under *much* more stressful conditions.

The .ogg library IV used may not be optimized for Windows? Or maybe they aren't going through DirectX Audio, and there are audio latencies which add up badly under stress? An explanation like this makes more sense (to me) than thinking the AI code runs slower under some OSes. With updated drivers, even the OpenGL layer should run decently independent of the OS (??) -- and rendering 1000s of DGs requires little more than texture mapping which consumes little CPU. You can see (above) that rendering a 1000 buildings doesn't tax the Windows OpenGL layer any worse than it does the Linux one. I dunno, but the audio subsytem seems like a reasonable cuplrit.

It would be interesting to see if silencing the DG sounds fixes the performance issues. I haven't tried it yet (just thought of it now), but going into sounds.txt you could either zero out the DG sample group listings, or tweak some of the Event parameters to try to reduce the processing load associated with DG sounds. If this works, then a slightly modified sounds.txt might enable *much* larger (or smoother) DG wars under Windows mods (or even the original game).

...Something like reducing the range of the DG sounds, so fewer of them have to be mixed at any given moment. Or changing all the "UpdateConstantly" parameters to "UpdateOncePerLoop" in the Darwinia section of sounds.txt. There are A LOT of UpdateConstantly parameters in that section. x1000 DGs...

[The GoldFish]

Why not compare the CPU usage %age for sounds between versions.

[brice]

Why not compare the CPU usage %age for sounds between versions.

I'm not sure how to do that reliably. Under Linux, Darwinia grabs nearly 100% CPU when started. Linux multitasking is responsive enough that a windowed Darwinia doesn't make the desktop *feel* like it's got 100%, but it always does.

I haven't checked CPU under Windows (I'd have to reboot), but I'd be surprised if it didn't grab 100% the same way.

The Darwinia engine obviously has it's own internal scheduler for using the available CPU. The performance problems under Windows would make the most sense if some OS service under Windows costs more than under other OSes. Audio makes sense, but that doesn't make it true. I already tried changing all the UpdateConstantly's to UpdateOncePerLoop for all DG Events, plus their lasers... and it didn't make any difference. I haven't tried nulling the DG sounds yet. Even if that worked, it wouldn't be practical unless you wanted a silent mod. What's Darwinia without squealing green DGs???

Why not compare the CPU usage %age for sounds between versions.

DOH!! You of course meant the in-game CPU usage under sounds options... but I of course was in my own world.

Yes, this does seem to show some significant differences between OSes on the same dualboot machine. However it's hard to interpret (for me anyway).

Under WinXP, Darwinia reports it is using the "DirectSound" driver, which sounds like DirectX. CPU jumps around constantly from 5-10% to over 20% which is in the red. Restarting Darwinia with and without RealTime effects enabled doesn't seem to make a tremendous difference. Memory seems to have a slow steady leak. Hardware3D is off. The sound CPU load remains jumpy even when on the world map. It doesn't seem to correspond directly to fps (but CPU %s can under-report decievingly even in real Performance Monitors under both OSes, due to sampling errors).

Under Linux, Darwinia never reports more than 1% CPU in sound options -- even when fps goes low. Memory seems to have a slow steady leak here too. There is no Hardware3d option ("unavailable").

These numbers were taken from the same level I used above in the OP. It has lots of DGs and 2x better fps performance under Linux. I ran the level until performance degraded to under 20 fps on both OSes.

It seems clear that sounds cost more on my Windows setup on my dualboot machine. But none of this proves this causes the 2x DG performance issues. We don't know how the reported in-gam CPU % is being calculated. Is it a hard number, or a fuzzy heuristic for troubleshooting purposes?