It’s been… nearly eight weeks since my last post and, I must confess, I’ve barely noticed the time go past. It’s been pretty hectic, what with a new flat, several cities to visit, work being crazy busy and Glasgow city centre proving to be fairly entertaining. Broadband having been installed last week (finally, after some fairly shoddy service by the first people we tried to get coverage from) I will be posting more regularly now, time permitting.

A quick recap of the last two months, including lessons learned:

• New flat in the city centre leading to many nights out.
• Sleep can be bypassed for large periods of time.
• It’s more than possible to avoid the internet for several months without getting overly antsy.
• 22Mbps broadband is ridiculous. Tricky to max out.
• If you shave your head, it will likely snow heavily the very next day. Sort of like a rain dance, but colder. And with less dancing.
• Prague food: the right mix of cheap, tasty and large portions.
• Prague booze: the right mix of cheap, tasty and large portions.
• Prague culture: the wrong mix of tourism, Western influences and the painfully obvious.
• The less people you have at a party, the more people (as an absolute number) that will pass out.
• My new computer (a 64-bit dual core athlon with 2Gb RAM and other silly goodness) is moderately better than my old computer (comparable to a Casio digital).
• To do lists are the bane of my life, yet I keep them compulsively.
• Although sleep can be bypassed, when the lack thereof hits, it hits hard.

Need sleep. Real posts soon.

I know, I know. You’re wondering why I haven’t been filling your lives with long winded articles about obscure attacks, or strange foods or music that no-one else likes. When, you’re surely thinking, will I find out what film Gary preferred out of the four he saw in January? (It was probably Jarhead by a whisker, to ruin the surprise). The reasons are threefold:

1. Work. I’ve been incredibly busy for the last few weeks. There’s nothing quite like a deadline to kill all creative and expressive urges; it’s been a case of going to work, coming home, and vegetating.
2. Going out. To stave off going nuts from the above, I’ve been going out a hell of a lot, not least of all being Mr Murray’s 21st birthday last weekend. I do implore you to search out the photos from that particular night. Not because of the incriminating pictures of Derek (which are sadly mostly private) but for the fact I look like I’ve had a stroke incapacitating half of my face in nearly all of them.
3. Moving. Preparation and pre-moving “smaller” items into my nice new flat has been taking up a fair chunk of my free time. I’ve yet to figure out how to get a drum kit from here to there, but it’ll be a fun challenge. Semantic web geeks please note that it’ll mean that my GeoUrl (anyone actually doing anything with those?) will be accurate within 5 miles for the first time!

Sadly, until I get broadband (and a phone line) set up in the new place, the silence will largely continue. See all five of you fine readers at the flatwarming, no doubt.

The piece on decompression bombs was not supposed to be a panic piece, as it seems is implied, rather it was an informative one about a hidden danger in handling compressed files and, in my view, a neat little trick.

To respond to some questions about it: someone asked how you would create the compressed file in the first place since compressing that much data would have to be done in memory, causing you the same problem. Very astute! The answer is that you don’t: the uncompressed data never exists. You need to know enough about your compression algorithm to construct the compressed file directly, writing the output without any real input. This is not that tricky for most formats. The fact of the matter is that you can download decompression bombs quite freely.

In response to a comment by David Russell, and to illustrate some points more clearly, I’m going to discuss some more concrete examples. First of all, David raises the issue that this just means “you restart”. Naively, yes, it does. A restart might not mean much to you, the home user, but it does to quite a few companies. Servers going down means a tangible loss in revenue and respect in the marketplace.

Moreover, denial of service attacks are used to blackmail companies. Mostly these come in the form of botnet attacks, but there’s no reason why a weak company couldn’t be taken down by decompression attacks. Frankly, without adequate defences, a decompression attack is far more effective than malicious pings.

Here are a few scenarios for you:

1. You are an admin for a company with a reasonable IT infrastructure. Being sensible, your mail server scans all incoming attachments for problems. You’ve also put the servers into a replication configuration so that if one server goes down then your mail queues redirect. Someone sends you a decompression bomb; your mail server goes down pretty quickly because the scan tries to handle it in memory. For every minute the servers are down, you lose X pounds. Depending on how much you’ve thought about this, X can be a large number. Your redundant servers can’t easily help because they’ll start scanning the file and falling over. You have a few solutions: you can switch off scanning (obviously dangerous). Maybe you can switch off replication, though that generally means isolating that queue and dropping the mail on the floor while you figure out what has happened, potentially costing more money.
2. You are an image storage/processing company who let your users upload images (like, for example, flickr). People upload whatever photos they like and let you display them. Because you want to save a considerable amount of bandwidth, you accept GZip encoded transfer of images. This saves you Y million a year (a conservative estimate for a large site). Someone sends you an image decompression bomb, which expands to some large amount on your servers. Your resources get drained while you either try storing it or sort it out. What can you do? Maybe you take the one time hit in writing it to disk; poor strategy if someone is using this as part of an effective distributed denial of service attack on your site. Maybe you just kill the GZip upload which, as we know, will cost you at least Y million a year; not a good idea if you’re looking to get promoted any time soon.

There are solutions to these scenarios (which I would like to leave as an exercise for my readers, comments anyone?), and I’m sure there are also more tricky scenarios for the really devious (again, I’d love to hear something quite sneaky). The point is that if you accept any content which is compressed, you should be aware of how this could affect you.

Solitude, as of today, has been running for three full years. Bloody hell. I’ve been going over a few random archive posts and it really is amazing how much the tone and style has changed over that time. I look back to early posts and see just how much my focus and enthusiasm has changed, I’m finding it much harder to post these days, particularly about technical issues which is bizarre since I understand technology and code far better now than ever. Half of the stuff I think about doing doesn’t seem as worthy of a post these days: I assume most of the people who read this would already know what I’m talking about and don’t need it again, and I certainly wouldn’t be writing it for myself.

As has become tradition on this site, last year I made predictions about what to expect on Solitude and got it hugely wrong. The Solitude sister project (code named “P”) failed to appear because I did literally nothing towards it. The post rate was light in the first half of the year, but got worse in the second half rather than better. Hooray!

Predictions for this year? More code. I was far too apathetic towards coding last year. Having done so much to finish my degree, then doing it as a full time job for the rest of the year, the last thing I wanted to do was come home and code more. However, in that time, the list of projects that I wanted to work on grew hugely (there are over 50 things on the list now). I hope that I’ll have time for at least some of them. Also, I will make an attempt to work on “P”. I think it would be good for me, a very different kind of website to Solitude. Of course, given past predictions, none of that will happen.

To anyone who has ever read this site, or who continues to do so despite its shortcomings, thanks. Hope you stick around for another year.

From random discussions I’ve had with people over the last few months, it seems to me that not too many people really know about a particularly nasty and cunning form of computer attack: the decompression bomb.

First, think about compressing data. Most of us have used an application like WinZip or some similar tool to make our files smaller, albeit less accessible temporarily. These programs take advantage of how signal to noise ratios work in most representations of data; that is, the way we try to represent information is almost always ineffecient. There are usually techniques available for removing the parts that aren’t so important, and letting important (or common) information take less space.

For example, the algorithm behind the ZIP programs is largely based on LZW compression: a technique that collapses common substrings down into single codes. Obviously, I’ve completely glossed over how it works, but if you want to know the information is out there (any computing scientist worth their salt should have at least an idea of how this sort of thing works).

Using a randomly selected document of, say, English text, you would expect a maximum compression ratio of 10:1. That is, given a 100k file, you would expect a zip file of around 10k. That’s pretty good compression! The language being used and the specific text will cause the compression ratio to vary greatly, but the principles remain the same.

The thing is, if you carefully construct an example document, you can get a compression ratio much higher. How much higher? MUCH, MUCH higher. For example, if you created a PNG image containing just one colour repeated over and over then you could easily get a 1000:1 ratio. For a text document containing 1 character repeated over and over, it’s possible to shrink 100Gb to about 6k. Think about that, it is a huge difference: 1.7e7:1.

That’s all well and good as an interesting experiment, but what does it mean for an average user? Imagine I had constructed one of those zip files that had shrunk 100Gb down to 6k and I sent you that file. If you trusted me, you might try to open it. Therein lies the problem: while you can readily accept the zipped file, the chances that you have the 100Gb of free memory (including virtual memory) to accomodate the decompressed file are bloody slim. When you try to open one of these files, your computer will quickly become overwhelmed and stop responding; all of the free memory having been used up, it can’t do anything else. You effectively suffer a denial of service attack.

That is what we call a decompression bomb.

There is another factor that could cause problems for people who are careful when opening files: well-meaning programs can open them anyway. If the file arrives on your system (either by explicit downloading or by, say, a mail program fetching it), it’s likely that anti-virus software installed on your system would then want to check if the file contained any viruses. To do this, it pretty much has to decompress the file in memory, leading to the same problem. Oh dear.

Most modern anti-virus software has some defences against decompression bombs, but they can still cause significant system lock-ups while figuring it out. Perhaps more evil is compressed web content, whether images or GZip encoded HTML. No modern browser has a strong defence against decompression bombs. Relatively small files (100Mb decompressed) are usually handled quite well, in that the browser doesn’t crash completely, but go much bigger and most systems will run into trouble. Because browsers, sensibly, accept GZip encoding by default, any URL can hide a problem.

Thankfully, problems don’t arise much in practice because there is little to be gained from this activity: if you take out someone’s system, you’ve annoyed them but you can rarely turn this to your advantage i.e. you can’t elicitly install spyware.

If you want to see some more figures or examples (at your own risk), then the AERAsec decompression bomb page is a great start. It’s where I got a few of my figures from (so thanks for that!), and has a link to some real examples you can try.

Now, Decompression Bombs Part 2 outlines some real-life examples and answers a few questions!