DEFCON launches, everything explodes
Posted: Wed Oct 04, 2006 11:56 am
On Friday 29th September Introversion Software launched it's third game DEFCON to an eager online crowd, and in the five days since that time we've been busy repairing everything that exploded. The last few days have been full of some serious ups and downs, and its about time we let everyone know what happened.
Launch day was basically an exercise in fire control. Someone remarked it kind of felt like manning a nuclear submarine deep underwater, trying to find a safe way to surface and deliver your payload while enemy carriers dropped depth charges on you from above (I suspect they’d been playing Defcon a bit too much.) Most of our web servers went down about an hour before launch time of 6pm GMT, and it took us quite a while to get things back up and running. The basic problem was simply that our services were totally overloaded. By the time our website was back up (minus forums since they were eating CPU power) the game client had already been released via various mirrors like FilePlanet, and of course the Steam version had gone live on schedule. We emailed activation keys out to all of our customers and pointed them at the available mirrors, and eventually got the main site back up and running. Suddenly users could click on our Download button directly, and we just sat and watched as the numbers went through the roof.
For the first time ever, Introversions bandwidth usage was measured in Terabytes.
The public service that lists all Defcon games is called the MetaServer, and it runs on a powerful dedicated server with a superfast internet connection. Hundreds and hundreds of game servers started springing up all around the world, peaking at about 2,000 simultaneous games on the night of the 29th. We’d stress tested the MetaServer in advance but were not prepared for this volume of usage. Our custom bit of software was having to handle 30,000 incoming messages every minute, meaning if you spend longer than 2 miliseconds thinking about one request you end up backlogging. This caused us all kinds of problems – steam authentication was failing, users couldn’t get server lists a lot of the time, and often the lists they received were out of date and contained servers that had long since shut down. We worked hard on improvements pretty much all night, fixing the worst of the problems and improving the performance. Some of these problems still affect the MetaServer now, although to a much less severe degree, and we’re still working on daily improvements.
Saturday and Sunday involved a typically massive Introversion meeting in the Flying Hamster (our codename for the Flat in Hamstead). Much was discussed, wine was consumed, and the entire company was restructured. We’ve been operating under a model we put together in 2002 when we had just 3 employees and only one project in development, and that model hasn’t been holding up well with our recent expansions. We agreed on the immediate development priorities – MetaServer optimisation, Mac Defcon, the first of many support patches for Defcon, and a couple of other top secret projects that we will announce in due course.
There is one other story of note from that weekend. Due to a fairly major communications failure, our internet service provider dismantled our web server on Sunday morning while it was still live. I have never personally heard so many swear words uttered in such a short period of time. This is just one of those crazy things that only happens when your website is the busiest it has ever been, and led to about six hours of downtime while we transferred the entire Introversion website to another server. What can we say, shit happens.
Monday was the Defcon Launch Party. We were slightly worried about topping the Darwinia launch party (which was essentially a really good house party in the Flying Hamster), but we’re not worried about that anymore. The Defcon Launch Party was awesome fun, and took place in the Cabinet War Rooms in Whitehall, London. For those that don’t know, the War Rooms are basically a series of underground bunkers and control rooms built by Churchill during the second world war - an extremely atmospheric venue, and I’d guess we peaked at around 150 guests. Here’s the Invite:
We had four player Defcon running on laptops at one end of the room, with the game projected on the wall for guests to watch. The laptops had Internet access so we had loads of guests logging in from outside to watch. At the other end of the room we had Wargames projected onto the wall. We had three DJs playing music all night (the same DJs as the Darwinia launch party) and all the key Introversion staff wore RAF uniforms. It was a very cool party that we won’t forget quickly. A big thank you goes out to everyone who came to the party, and we hope you enjoyed it as much as we did.
If you have some pictures of the party on your cameras, please send them to us! Please email them to defconlaunchparty@googlemail.com. If you could reduce them to sensible file sizes first, we’d appreciate it. 1024x768 is more than enough resolution per picture.
Sales of Defcon since launch have been excellent, with about one third choosing to buy from us and get their free boxed copy, and two thirds opting for the convenience of the Steam version. It’s looking like Defcon is going to be our best selling game yet. We’ve got a truly massive backlog of boxed copies to send out, so please accept our apologies if it takes slightly longer than normal. Interestingly, sales of Darwinia and Uplink have also come up quite a lot, with many customers choosing to buy pretty much everything we sell in a single transaction. This makes us very happy We’ve produced a soundtrack CD and a couple of posters, and they’ll soon be added to the store with more details at the time.
We’re amazed that people have begun to mod the game already, despite Defcon having pretty limited modding ability to start with. We’re planning to improve on this with future patches. So far most of the mods have been graphical changes, but we’re confident some new maps will start to appear soon. We’re going to open up a couple of new Forum channels : one for defcon mods to be advertised and discussed, and another for people to discuss ideas for improvements that can be made to the game. The core gameplay is going to remain locked down now, but there is lots of room for expansion of the core idea.
Finally, we should mention the reviews. It’s looking like we’re picking up 80 – 85% in most reviews, which we’re thrilled by. There are loads to choose from and many more on the way, but we’d particularly like to direct readers to the PC Gamer review in the UK (87% and the MUST BUY award), and the Eurogamer review (8/10).
Thanks!
Launch day was basically an exercise in fire control. Someone remarked it kind of felt like manning a nuclear submarine deep underwater, trying to find a safe way to surface and deliver your payload while enemy carriers dropped depth charges on you from above (I suspect they’d been playing Defcon a bit too much.) Most of our web servers went down about an hour before launch time of 6pm GMT, and it took us quite a while to get things back up and running. The basic problem was simply that our services were totally overloaded. By the time our website was back up (minus forums since they were eating CPU power) the game client had already been released via various mirrors like FilePlanet, and of course the Steam version had gone live on schedule. We emailed activation keys out to all of our customers and pointed them at the available mirrors, and eventually got the main site back up and running. Suddenly users could click on our Download button directly, and we just sat and watched as the numbers went through the roof.
For the first time ever, Introversions bandwidth usage was measured in Terabytes.
The public service that lists all Defcon games is called the MetaServer, and it runs on a powerful dedicated server with a superfast internet connection. Hundreds and hundreds of game servers started springing up all around the world, peaking at about 2,000 simultaneous games on the night of the 29th. We’d stress tested the MetaServer in advance but were not prepared for this volume of usage. Our custom bit of software was having to handle 30,000 incoming messages every minute, meaning if you spend longer than 2 miliseconds thinking about one request you end up backlogging. This caused us all kinds of problems – steam authentication was failing, users couldn’t get server lists a lot of the time, and often the lists they received were out of date and contained servers that had long since shut down. We worked hard on improvements pretty much all night, fixing the worst of the problems and improving the performance. Some of these problems still affect the MetaServer now, although to a much less severe degree, and we’re still working on daily improvements.
Saturday and Sunday involved a typically massive Introversion meeting in the Flying Hamster (our codename for the Flat in Hamstead). Much was discussed, wine was consumed, and the entire company was restructured. We’ve been operating under a model we put together in 2002 when we had just 3 employees and only one project in development, and that model hasn’t been holding up well with our recent expansions. We agreed on the immediate development priorities – MetaServer optimisation, Mac Defcon, the first of many support patches for Defcon, and a couple of other top secret projects that we will announce in due course.
There is one other story of note from that weekend. Due to a fairly major communications failure, our internet service provider dismantled our web server on Sunday morning while it was still live. I have never personally heard so many swear words uttered in such a short period of time. This is just one of those crazy things that only happens when your website is the busiest it has ever been, and led to about six hours of downtime while we transferred the entire Introversion website to another server. What can we say, shit happens.
Monday was the Defcon Launch Party. We were slightly worried about topping the Darwinia launch party (which was essentially a really good house party in the Flying Hamster), but we’re not worried about that anymore. The Defcon Launch Party was awesome fun, and took place in the Cabinet War Rooms in Whitehall, London. For those that don’t know, the War Rooms are basically a series of underground bunkers and control rooms built by Churchill during the second world war - an extremely atmospheric venue, and I’d guess we peaked at around 150 guests. Here’s the Invite:
We had four player Defcon running on laptops at one end of the room, with the game projected on the wall for guests to watch. The laptops had Internet access so we had loads of guests logging in from outside to watch. At the other end of the room we had Wargames projected onto the wall. We had three DJs playing music all night (the same DJs as the Darwinia launch party) and all the key Introversion staff wore RAF uniforms. It was a very cool party that we won’t forget quickly. A big thank you goes out to everyone who came to the party, and we hope you enjoyed it as much as we did.
If you have some pictures of the party on your cameras, please send them to us! Please email them to defconlaunchparty@googlemail.com. If you could reduce them to sensible file sizes first, we’d appreciate it. 1024x768 is more than enough resolution per picture.
Sales of Defcon since launch have been excellent, with about one third choosing to buy from us and get their free boxed copy, and two thirds opting for the convenience of the Steam version. It’s looking like Defcon is going to be our best selling game yet. We’ve got a truly massive backlog of boxed copies to send out, so please accept our apologies if it takes slightly longer than normal. Interestingly, sales of Darwinia and Uplink have also come up quite a lot, with many customers choosing to buy pretty much everything we sell in a single transaction. This makes us very happy We’ve produced a soundtrack CD and a couple of posters, and they’ll soon be added to the store with more details at the time.
We’re amazed that people have begun to mod the game already, despite Defcon having pretty limited modding ability to start with. We’re planning to improve on this with future patches. So far most of the mods have been graphical changes, but we’re confident some new maps will start to appear soon. We’re going to open up a couple of new Forum channels : one for defcon mods to be advertised and discussed, and another for people to discuss ideas for improvements that can be made to the game. The core gameplay is going to remain locked down now, but there is lots of room for expansion of the core idea.
Finally, we should mention the reviews. It’s looking like we’re picking up 80 – 85% in most reviews, which we’re thrilled by. There are loads to choose from and many more on the way, but we’d particularly like to direct readers to the PC Gamer review in the UK (87% and the MUST BUY award), and the Eurogamer review (8/10).
Thanks!
).
Glad to hear things are going well - you deserve it!
...