Friday, 22 June 2007

Fiddling around and the simple storage solution

I was invited on to the Amazon Computing Cloud yesterday - a seminal moment as I have been desperate to have a go on it for weeks now, but disaster! I was passed over because I had foolishly forgotten to open an S3 (simple storage service) account which is a prerequisite, despite my grovelling and the immediate opening of an S3 account I appear to have missed my window of opportunity.

That said it has given me a little time to investigate the simple storage service and in itself this has opened a few doors in my mind. Storage in the S3 bucket is very cheap indeed at 15 cents per Gig per month, this means for example that I could store my largest SQL database, at 2Gig and rising, on the ultimate storage system for 14p per copy per month with transfer costs of 10p. Doing a little simple maths therefore means that backup scenarios would cost:
  • 24p per month per copy
  • £1.78 per copy per month
  • £50 per month for a rolling 30 day backup
  • £650 pa. for a daily backup rolling on for 12 months (almost 800Gb of data)
The real magic here is not that I can get someone to host almost a terabyte of data for £650 pa but the infrastructure behind it, storage spanning multiple data centres in multiple countries, is fantastic. There are only 2 slight problems, there is a file size limit of 5Gb at present which hopefully will have increased by the time my bloaty database has got anywhere near that, also my backup software Red Gate does not at present support S3.

So I'll have to write a little program one of these days to bounce my off site backup from our Manchester server over to the S3 cloud on a nightly basis then I really, really will have an offsite backup I can be satisfied with. Alternatively our friendly Red Gate developers might like to take the hint, get an S3 account and crack their programming type knuckles. Utility computing has to be the future of off site backup so get in there, I wont charge for this revolutionary piece of advice but I have broken my promotional Red Gate pen so if someone wants to send me another one we'll call it quits :o)

Whilst I am on the subject of S3 there is a fantastic little add on for FireFox which I found very useful for getting things moving, the S3 organizer, its a good job the programming is better than the spelling of organiser :o)

And speaking of really useful little pieces of software anyone who does any programming of web apps should get themselves a copy of fiddler. Its a really rather sweet little program written by Microsoft (tun tun tun!) and distributed free! Maybe Bill finally has enough.

Anyhow before I get on my soap box, fiddler is a very simply http proxy server which gives you a real time readout of what traffic is moving through your Internet connection. As we are using Flash couple with AmfPHP we constantly struggle to debug in an efficient way but not anymore! I wont spoil compadre Robs review right now but just get it and watch out for the bug eyed review.

Wednesday, 6 June 2007

In the queue for the ECC

There have been lots of things going on over here for the past month but nothing really blog worthy, aside from this, having a baby has somewhat curtailed my late night blogging sessions, so this is the first post since May 10th!

We have signed up for the ECC, this is not another odd-ball European thingy but in fact Amazon's new web service christened the Elastic Computing Cloud. The concept is what is known as utility computing in that you create a new Virtual Machine and only pay for the storage and processor time you use. This means that within reason your online application can scale wonderfully from being a dusty corner of the web which no-one ever uses to the latest craze with millions of users in minutes and your server wouldn't crash (as long as your code is well written of course).

We have a little online application which we will very shortly be polishing for general release and this seems like a great opportunity to keep our initial installation costs very low but have the ability to scale quickly to meet the needs of our new users :o) Then of course, when things have settled down, and we know what sort of power we are going to require long term we can make a more informed decision about buying our own hosting kit without having to wade through goat intestines with the help of a good soothsayer.

The spanner in the works of course is that everyone else want to make use of this wonderful new service as well and I have found myself in the queue. This seems to be a rather annoying trend in fact, I queued for Joost, I queued for Google applications for Domains and now I am in another queue. I suppose it allows companies to test their systems without having a big embarrassing launch followed by teething trouble but I want it now :o(

I have looked for alternatives but it seems no one else is offering such a simple, well supported and dare I say cheap service within the means of the average web applications developer. Until now utility computing was in fact the preserve of the scientist wanting to test his quantum theories or analyse what happens in the middle of a cosmic jam doughnut (think SETI) so it seems Amazon are possibly on the cusp of having a runaway success. Every applications developer who doesn't want to ask his boss for a new server and equally doesn't want potentially unstable code on one of his precious live servers will want an account for testing stuff.

I am only surprised that Amazon beat Google to it as its just their sort of thing, I look forward to using the gutility computing cloud in about 2 months (probably for free) and inevitably the Microsoft computing cloud in about 2 years which will be compliant with the utility computing standards which the IEEE will have created and ratified by then but with of course.... Microsoft extensions.

A view from the rack is the personal blog of an IT manager who works for a pub company - hence beer