My friend George pointed out a link to an Infoworld article about a new startup, If this then that

I've been looking at internet-wide event processing for a little while. My feeling is that messaging and event-processing have been big and useful in enterprise IT for a while, but that so far there hasn't been much on the wider web ecosystem. And we are increasingly generating and consuming events - facebook status updates, google+ updates, location data, share prices, real-time sports data. So my hope has been that this is an area that can grow up and offer us some great new capabilities in the next few years.

So if this then that looks interesting. Unfortunately, it appears their site is down right now, so I need to follow up further, but my initial impressions from the infoworld article are

1. This is cool, and not too far from the lines I was thinking - I hope it generates some interest.
2. It reminds me a little of Yahoo's pipes, which was cool, but never really went anywhere. Albeit an event-based version, so as SQL is to CEP, pipes is to this I guess, in my mind.
3. That it relies on ifttt people hooking into your site. I hope I am wrong about this - it would be a shame for me to have to build a userbase of tens of millions of users before it's worth their while hooking into my API.
4. That it's proprietary, and that's reasonable given the amount of IP in those integrations. But then it depends on their integration with various APIs, and your event-processing remains locked in their system. This is a shame, and that's why I hope that underlying protocols for event dissemination get standardized, so that systems like this can become commoditized.

Anyway, these are just my thoughts, and ifttt seems like an exciting step increasing general excitement around events, and hopefully a baby-step to a brave new world where consuming, processing and emitting events across sites seems as commonplace as looking at a blog in google reader.

I acknowledge that adding something to the sidebar of my Posterous blog is not going to set the world on fire. The real meat here is that I have created a nascent library to pull data from the Hacker News website and I'm interested to see what all the creative people that hang out over there do with it. The code is available on github - help yourselves.

If you interested in software, or startups and don't already read Hacker News, you should. It's what reddit was before it got big, and what slashdot was before it got big before that. Fortunately, Paul Graham know what's what, so Hacker News has proved more robust.

I've spent quite a bit of time following articles on Hacker News, upvoting the ones that I like, and know I'll want to read again one day. It occurred to me that it would be awesome to make the stories I've upvoted available as a side-bar on this site. In this day of RSS that should be easy right? Wrong. Hacker News is deliberately basic, partly due to PG's time-constraints, and partly I suspect because keeping it basic means only people actively interested in the subject matter go there. As a result, it doesn't have an RSS feed for the list of your saved stories that it maintains. And Posterous doesn't seem to have any option to add an RSS feed to a side-bar anyway.

Architecting and building this little monstrosity

I have a couple of virtual private servers that I keep around to perform tasks like this. The plan here was to have a VPS periodically scrape the Hacker News site, and generate a static HTML file, which could then be included in the Posterous sidebar.

My choice of language for tasks like this is Python. There are a stunning number of libraries built into it, and it's easy to augment its power using packages from the PyPI repository. I'm sure that Ruby and Perl would be similarly capable, but this is definitely a task best suited for a scripting language.

I chose to use the Mechanize library to control interaction with the Hacker News site. Mechanize provides you with a 'browser' object that can be easily told how to fill in forms, or to click a link with particular text. This relieves the strain of having to keep track of state between calls to an HTTP library, and having to parse the returned HTML. This made it a lot easier to handle the stream of hidden forms and clicks that have to be done to log in to Hacker News using clickpass.

The library wraps around the Mechanize browser object, allowing the library user to easily log in to Hacker News (currently only available using a google login - patches/pull requests welcome if you'd like to contribute). Once logged in, the user uses the nav_* functions to navigate to the appropriate page, and the get_current_page_stories function to get a list of dictionaries, each item representing a story. An example interactive session might look like this:

>>> import hnscraper      
>>> h=hnscraper.HNScraper()
>>> h.login_with_google('USERNAMEGOESHERE','PASSWORDGOESHERE')
>>> h.nav_saved_stories()
>>> stories=h.get_current_page_stories()
>>> list(stories)[0]
{'score': 254, 'link': u'http://adgrok.com/why-founding-a-three-person-startup-with-zero-revenue-is-better-than-working-for-goldman-sachs', 'user': u'cjg', 'title': u'Founding a startup with zero revenue is better than working for Goldman Sachs', 'ord': 1, 'comhead': u' (adgrok.com) '}
>>> h.nav_more()
>>> stories=h.get_current_page_stories()
>>> list(stories)[0]
{'score': 61, 'link': u'http://www.aaronstannard.com/post/2010/06/12/The-Myth-of-the-Single-Person-Startup.aspx', 'user': u'Aaronontheweb', 'title': u'The Myth of the Single-Person Startup', 'ord': 31, 'comhead': u' (aaronstannard.com) '}

The get_current_page_stories function uses the BeautifulSoup library to parse the currently open web page. BeautifulSoup provides a pragmatic interface for pulling data out of HTML pages, allowing access through the tree of tags using dictionaries and lists, which allows you to quickly drill down to the data you need. For example, to get all the <td> tags for a particular row:

>>> s.findAll('tr')[49].findAll('td')</div>
<div>
<div>[<td align="right" valign="top" class="title">16.</td>,</div>
<div> <td><center><img src="http://ycombinator.com/images/s.gif" height="1" width="14" /></center></td>,</div>
<div> <td class="title">Cold Calling Techniques<span class="comhead"> (businessballs.com) </span></td>]</div>
<div>

As there is no class assigned to indicate which table rows represent stories, the get_current_page function naively looks through all table rows in a page, trying to build story dictionaries for those that are in a plausible format. This works pretty reliably in practice.

Building a page for Posterous to include

Martin Ruiz has a great blog post on how he uses a similar technique to add items to the sidebar of his blog. He uses a dynamicly generated webpage, whereas I build a static webpage to serve - in the event that I hit slashdot, maybe this saves my server from melting. 

The page itself is included in Posterous using an iframe added to my custom Posterous template:

<section id="hackernews">
<h1>Articles I've liked on Hacker News</h1>
<div class="sidebar-block">
 
</div>
</section>
 

The key thing to remember is that the included page does not inherit any of the CSS from the page importing it. To get the look and feel right, I opened up my posterous home page in Chrome, and fired up the Developer Tools. I can use this to see what CSS is getting applied to each element on a page, and replicate that in the page I will be generating.

I used the mako template library to build the final output - this is very similar to using a templating engine to dynamically generate content. I built a script, make-html.py, to drive the HNScraper, pass the scraped data to Mako, and save down the output ready to be served.

The End Result

You can hopefully now see the items I've found interesting in the sidebar on my blog. The fantastic libraries available for Python meant that I just had to write a bit of glue code to get this working - 170 lines in total. Working out how to get Mechanize to push through clickpass took a little time, as did getting the parsing right on anonymous-looking tables using BeautfulSoup, but all-told, I've probably spent longer on this blog post than writing code.

As I said, I've made the code available on github. I hope you find it useful, and I'd love to see what mash-ups people can use it for.

There you go: in time-honoured blogging fashion, I've used a highly inflammatory title to get your attention. Hopefully now I'll have got that out of my system, and the rest of this post can be reasoned and insightful.

A friend drew my attention to this animation at Alex Popescu's excellent myNoSQL blog. It features a pig and bear (?) arguing the merits of relational versus NoSQL databases. The bear is a NoSQL fan-boy, parroting lines such as "shards are the secret ingredient in the web-scale sauce - they just work". The pig condescendingly suggests /dev/null may be a better place to store data, performance-wise, and rants that he will move to farm rather than work in the same industry as the bear. The debate is not moved forward.

Now, the relative merits of relational and NoSQL databases is well trodden ground by now. Relational databases are reliable and proven versus the scalability and flexibility offered by schema-less NoSQL databases. Each has their place. There are some situations where reliability, and transactions, and all the other relational goodness is important - bank accounts are the standard example. Conversely, if you need to scale cheaply, and no-one cares all that much about a few missing tweets, likes or status updates, then NoSQL is the obvious solution. There are gray areas in the middle, where either or neither may be appropriate. But it is hard to argue that there is no place for one or the other, and you'd have to be a moron to make that choice (regardless of title of this blog post).

So why the animosity between the relational database crowd and the NoSQL crowd?

After all, there are plenty of areas where there is more than one solution. No one argues vociferously that I must use Google or Bing for all my searches - sometimes one is better, sometimes the other. Sometimes Excel is a more convenient tool for analysis, sometimes Matlab would be a better choice, and everyone seems happy with that.

Although the argument is often framed in terms of facts - "MongoDB is web scale", "You might as well write to /dev/null" - I think it is more telling to look at the interests of those making the arguments. Well, one difference between the examples above and database technologies is that there is a group of people with a vested interest in relational databases: DBAs.

I suspect that DBAs, en masse, think that NoSQL databases will ruin their livelihoods with their lack of schemas and tables to CREATE and UPDATE. On the other side of that, web developers think they would like to cut out the bureaucracy of endless CREATE and SELECT statements as they make web applications, and as extension of that, not having to defer to a DBA on matters of database style, and schema-less databases seem to promise that.

I understand the defensiveness, and I understand the desire to make dealing with data easy. But I think both of these arguments are wrong.

Dealing with large amounts of data has never been easy. DBAs are valuable not because they now they know every clause a SELECT statement can have, but because they understand data, and particularly the data under their care. They know what indexes are going to help performance. They know how to defend against the application layer inserting junk rows with inappropriate NULLs, or against application programmers who want to make a column called "COL_1" to contain comma-separated data because it would just be convenient. A good DBA can free developers from thinking about data-quality issues, and act as a multiplier on development productivity.

This is not going to go away just because some applications run against a MongoDB back-end, rather than a MySQL back-end. Sure, the commands will be different, but the skills will not, and much of the knowledge and experience transfers.

In short, I think data volumes are exploding, both for relational and non-relational databases, and the DBA skill set will be in higher and higher demand over the next few years. Hopefully, when this comes to pass, we can stop the defensiveness, stop the fan-boyishness, and just use the best tool for whatever job we need to do.

Or we can use CSVs and VBA.

Update: My friend George has written a thoughtful response to this post, arguing that NoSQL solutions will benefit from standardized query languages, and query planners, and that it may be as easy to adapt existing solutions to be distributed as it is to re-invent the wheel. 

Occasionally it's fun to get away from "serious" programming and do something just for fun. Serious programming these days tends to require using several, possibly tens of external frameworks and libraries. That's fine, and you get a lot more bang per line of code that way. But it can be fun to go back to basics, and make something from scratch.

With that in mind, I've been looking at the JS1k competition. The idea is to do something interesting in under a kilobyte of JavaScript. The great thing about this is that you can get started with just notepad and a browser. The new canvas element in HTML5 makes it surprisingly easy to do interesting graphical effects in JavaScript. But it requires some ingenuity to get the size down, particularly any data you might want to use is probably better generated procedurally.

My effort is a spinning tetrahedron. I enjoyed Steven Wittens description of making his demo, so I thought I might describe one aspect of mine here.

My initial effort ran to over 2k, so I looked at a few techniques to get the size down. Some were pretty obvious: changing variable names to be single letters helped. Inlining functions further reduced the size to 1.3k - each function requires you use at least "function" and "return", but you also get quite a lot of redundancy assigning the results to a local variable for example.

One method that helped, and is a bit more interesting, was to generate the data, rather than store it. The data in the case are the vertices and the line list [lines from vertex 0 to vertices 1, 2, 3, from 1 to 2 and 3, and from 2 to 3]. Taking the vertex list first, one way to represent it would be as a list of lists, with each inner list having 3 items representing the x,y and z coordinates, using 65 bytes:

P=[[-0.5,-0.5,-0,5],[-0.5,0.5,0.5],[0.5,-0.5,0.5],[0.5,0.5,-0.5]]

Accessing a coordinates using something similar to "P[i][0]" uses 7 bytes, and this will be done once for each of the x, y and z coordinates for a total cost of 86 bytes - a not insubstantial portion of the 1024 allowed.

A better way is to not explicitly store the data, but to try and use bitwise operators to calculate it. Looking at the first coordinate, we can see that it is that it is -0.5 for the 0th and 1st vertices, and 0.5 for the 2nd and 3rd. I notice that 0, 1, 2 and 3 in binary are 00, 01, 10 and 11, and that the second digit, less a half, would be precisely what I want. So I can replace "P[i][0]" with "(i/2&1)-0.5". Similarly, the second coordinate is the first digit, minus a half, and so "P[i][1]" can be replaced with "(i%2)-0.5". The final digit is a little harder - it is 0.5 when i is 1 or 2. One way to think of this, is that it is 0.5 when precisely one of the binary digits is 1 - so this sounds like a job for xor. Sure enough, the final coordinate can be represented by "((i/2^i)&1)-0.5". Each of these expressions is longer than using the indexing operator on P, but it is now possible to throw away P. In fact, using this representation uses 34 bytes rather than the 86 used by explicitly creating the vertex data.

Similarly, the line list does need to be stored explicitly. In a pyramid, every vertex is connected directly to every other vertex, so it is sufficient to use a double-loop over the vertices "for(i=0;i<4;++i)for(j=i;j<4;++j)".

With a tetrahedron, these savings a relatively small. For a cube, with twice as many vertices, they are larger: the vertex list can be stored by using the jth binary digit to represent the jth coordinate, and the line list can be represented by double-looping over each pair of vertices again, but only drawing those where exactly one binary digit differs [ie vertex a and b are connected if a^b is a power of 2, which is true if (a^b)&((a^b)-1) - check bit-twiddling hacks for more bitwise goodness]. For an icosahedron, it is not immediately clear what a good representation 

I just had my first ride on a Barclays Cycle Hire bike. Here are some random thoughts on it:

I think they should be called Barcycles. I like portmanteaux.

It's less scary riding in London than I thought. I used to ride occasionally in Cambridge, but mostly on back-roads, so I always found the thought of cycling in London pretty terrifying. While I chickened out of joining 3 lanes of traffic on Euston Road on my first ride, cycling with traffic wasn't as scary as I expected - I expect that traffic is more wary than usual of people on.

The bikes themselves are pretty good. Perfectly pleasant to ride for half an hour or so. I've never driven a carbon-fibre featherweight bike, so I don't know what I'm missing, but I have walked a long way in London, and these bikes are definitely faster than that.

FipLab's London Cycle: Maps & Routes app is cool. I looked at a few contenders, and all seemed reasonable. FipLab's effort seemed best. I just sent an email to the company, and they responded in a minute, letting me know that they are looking at colour-coding the pins based on bike/space availability in the near future. Awesome.

There are a few organization teething problems. I couldn't find a docking space at the dock nearest my work. I understand that there is a large inward flow of bikes in the morning, and and a large outward flow in the evening. I hope that this will get dealt with in time, as demand stabilizes, and TfL can better plan how much redistribution is needed. In the meantime, it's a little frustrating to have to walk back from a distant cycle station.

Very positive for cycling in London. This makes it a lot easier to cycle in London. No need to find somewhere to keep a bike now. More importantly, I think it stands a good chance of creating a virtuous circle which fixes the lack of cycle routes in London. More people cycle, so more cycle routes are created, so more people cycle. 

Barclays got themselves a good deal. Put in £25m over 5 years, less than half the estimated cost for the first year of the scheme, but get their logo and colouring plastered all over London. I've no objection to the corporate sponsorship - if anything it's positive - but Boris's team could have pushed for a better price.

Cycle shops could take better advantage. I went to a couple of shops to look at buying a helmet. I asked the very helpful guy at Cycle Surgery if there had been many people using the Barclays Cycle Hire scheme coming in for helmets - he didn't know. With 32,000 members in the first week, that has to be the biggest upsurge in cycling the capital has ever seen. Ok, so not many of these people are going to spend £1,000 on a bike frame, but I would expect that most of these people would value the contents of their skulls at more the £50 a reasonable helmet costs. The cycle stores could pushing this. Stick a board outside the store proclaiming that you sell helmets for people using the scheme. Pay people to hand out flyers by cycle stations offering a discount on helmets, or even just describing what you have. But it seems like they are blithely ignoring it.

Be interested to hear of your experiences, or any hints and tips to get the most out of the scheme.

I just read Jason Cohen's advice on the Smart Bear blog about how to pick a company name. It's excellent advice, and I wish that article had been written before I was trying to come up with a name for Cobalt Quantware last year. It also reminded me that I wanted to write a blog post about the process of coming up with a product name, which I have just been through.

The core of Jason's advice is that the name is a lot less important than what the company does. I hope that is true, and I think it is. It is good advice, because it takes the pressure off you to think of the perfect name. Nonetheless, when you fire up your IDE or editor of choice to create the new project or solution that will be your product, it will ask you for a name. So you shouldn't put pressure on yourself to come up with a perfect name, but you do need a name.

Don't: Use a half-assed Product Name

My first instinct was to use the first name that came into my head, and hope that I thought of something better later. I spent about 3 minutes coming up with the name. This was a mistake for several reasons, and so I'd advise you to avoid doing that.

I was never really keen on the name. This reduced my engagement with the project. Not by a huge amount, just marginally. Nonetheless, trying to bootstrap a new business and create a product is hard. It is not wise to throw away motivation.

Customers and advisors were not keen on the name. One of the repeated comments I would get is that the product looked useful, but the name wasn't great. Customer and advisor feedback is invaluable. It is a waste of both your time for you to explain why you have a crappy name that you intend to change. Worse, it is a wasted opportunity if they are telling you something that you already know. Far better to fix it, and hear about the feature that they really wish was there that hadn't even occurred to you.

Do: Use a codename if you can't come up with a Product Name

Later, I adopted a codename for the project. A codename is just a cool-sounding word that is a temporary placeholder for the final name.

I decided to use names of British railway stations for codenames, using a station with a cool-sounding name beginning with "A" for the first codename, "B" for the second codename, and so on. There are a hundred themes you could use: cool-sounds birds, cool-sounding cities, cool-sounding metals etc.. This can be done pretty well without getting others involved, and without taking a large amount of time.

When I did this, I started getting better quality feedback. Customers knew that this wasn't the final name, so it wasn't up for discussion. We could move on to discussing important topics, like what they wished the product would do.

Do: Come up with a Product Name as early as possible

As well as a solution/project name for the codebase, you may also need a name for JIRA, or other bug tracking software, for a wiki, for documentation. Some of those places can be easily changed. Some less so: you can't easily change the project code used by JIRA for emails, for example. The sooner you come up with a final product name, the fewer places you will need to change, and the fewer headaches you will have making those changes.

Later, but before release, you will need to start creating marketing materials - a document describing your product to send to potential customers, for example. At this point, you pretty much need a final product name. Your product name will be in large type at the top of that document. You don't want an asterisk next to it.

Eventually you will release your product, and at that point you will be pretty irrevocably committed to the name you have used. Yes, products change their name, but they lose a lot of brand recognition when they do.

Do: Get a group together to brain-storm for Product Name candidates

I naturally tend to gravitate towards solving problems on my own, rather than in a group. I tried to do that with the problem of coming up with a product name. That worked badly. I partially filled a whiteboard with suggestions that I wasn't happy with. One of my advisors suggested than in a large company she had worked for, they would typically gather a group to brain-storm for product names. That was great advice - this is a type of problem that screams out for group brain-storming.

In brain-storming, you have two aims: to suggest ideas that could be good solutions, and to suggest things that could trigger ideas in others. It's important to create a positive environment where people feel free to suggest anything, no matter how off-the-wall, and it's important not to criticize or evaluate ideas at this point. There are plenty of good resources on brain-storming on the web. I'd recommend businessballs.com for this and other topics.

One problem for me was that I'm a sole-founder, so I didn't have anyone to brainstorm with. I solved this by gathering a group of friends in a meeting room I hired for a couple of hours over a weekend. I traded with them the promise that I would owe each of them a couple of hours (but that I'd appreciate it if they didn't choose tasks that would make me cry).

I decided on a group size of 6, as being large enough to bounce around ideas, but small enough not to be intimidating. I deliberately chose friends that didn't know each other. This helped us focus on the task, rather than pal around, and it maximized the variation in suggestions. We also had lunch together afterwards. I think everybody had a good time. Common friends, unsurprisingly, got on well, and people enjoyed the unusual exercise.

We generated several hundred suggestions, many of which I was very happy with, and all it has cost me so far is building a barbeque and helping a friend move.

Do: Filter down to a shortlist

I transferred the several hundred suggestions from flipcharts to a word document. From the several hundred suggestion, I filtered through several times, first down to about 100 names, and then to a shorter list of around 25, and finally to 10 or so names. I did this at different times, to ensure that the names that made it through weren't purely based on a particular whim at the time.

I also found that some suggestions were fragments - either a good descriptive name for the product, or a cool-sounding adjective that could be incorporated into a product name. I tried to save these as I filtered, and combined them to form a few extra names for the shortlist.

I checked the shortlist to ensure that I wasn't infringing on trademarks, and that the names were not already in use, ideally for any product, but certainly for similar products, or other products that our customers might already use.

In the final filter, from 25 to 10, I did not filter purely on quality - I was pretty happy with most of the 25 - but as well to ensure some variation in the final set. I would rather get feedback on more variable set of names, than a higher-quality set of very similar names.

Do: Get feedback on the shortlist

I gathered feedback from about 50 people over the course of a week or so, including customers and potential customers, advisors, friends, family, people I had just met, and so on. Everyone has an opinion on your shortlist, and most people are happy to give it.

The easiest way is just to ask if they'd like to see the shortlist of names for your product (you were talking about your product anyway, right?), present them with the list, shut up and make notes. Do shut up, because people will often volunteer things like liking one part of one name, and one part of another, which they won't if you just ask them which their favourite is. Do make notes, because you won't remember what someone said about 10 different names over dinner. In a pinch, I have recorded people on my phone, but people are generally a little more reticent to be recorded.

When you get back to base, systematically organize the feedback. Hopefully, it should quickly become apparent that there are a few names that people are responding positively to. Remember, these are being positively evaluated with respect to the names that you preferred out of a large set.

Final Thoughts 

The process of coming up with a name was not actually that time-consuming. In hindsight, it is pretty obvious too. But coming up with a process that worked well for me was a little trickier. In particularly, the key realization was that as a sole-founder I would need, and could get, others to help on this task. I hope that if other people are stuck on this task, or on other tasks, they can use my "mini-crowdsourcing" experiment as inspiration.

For me, having a final product name that I was happy with, and my customers liked, was a big punch-the-air happy moment. There was a huge contrast: Not having a product name was an increasing source of anxiety as we moved towards launch. Having a product name really renewed my energy and engagement towards the project.

I'm really excited about Cobalt Quantware again, and about our first product. We currently in stealth mode, but I look forward to filling you in later, at an appropriate time.

NRN = No Response Necessary

A few years back, I was thinking about learning to fly. Other things got in the way, and it didn't happen, but it's something I may go back to in the future. That didn't stop me massively geeking out around flying though. One of the things I liked to do was listen to air traffic control online. It's actually quite relaxing as background noise, in the same way that listening to the shipping forecast is. I highly recommend http://www.atcmonitor.com/, where you can see live radar around Atlanta as well as listen to the air traffic controllers.

One of the things I've noticed is that air traffic controllers and pilots don't waste a lot of time when talking on the radio. They need to say what needs to be said, and then stop talking. They are busy, and need to focus on flying and directing planes, and there is probably another person who needs to transmit a message.

It's quite common for them to finish a transmission with "Good day". The implication is that they have nothing further to say, and aside from the other party repeating what they have just said to confirm, they do not expect to speak again. They are not being rude, but busy, and considerate of the next guy who needs to speak.

In a lot of ways, it would be nice to have a similar convention in other avenues of life. Sometimes emails seem to bounce politely backwards and forwards, long after everyone has run out of things to say.

So I'd like to propose inserting "NRN:" in the subject line of an email to indicate that the other party is not obliged to respond. It's short for "NO RESPONSE NECESSARY". The idea is that I won't be offended if you don't respond.

So if I you send me an email asking a question, and I give you an answer, I could stick "NRN:" in the subject line. You are relieved from any duty you might have perceived to send me a thank you email. And you don't have to flag the email for further response. You can if you want, or have something else you need to say, but don't feel obliged. Don't worry - we are done here and everything is fine.

In a previous blog, I wrote how excited I was by ZeroMQ, a very lightweight brokerless messaging system. I wanted to also try RabbitMQ using C# on my Windows development machine, but couldn't find any step-by-step introduction to get me to the point of running my first program. I'm there now, so I thought it would be good to remedy that.

I'd like to note that I'm not a RabbitMQ or messaging whiz. What you see here may or may not be best practice. If it's not, hopefully someone will chime in with some comments to put me back on the right path.

Here are the steps I took to install RabbitMQ and create a couple of simple C# clients to check that everything was working.

1. Download the Windows bundle zip file from the RabbitMQ download page. The current version at the time of writing is 1.8.0. It contains the Erlang installer, the RabbitMQ server, and bindings for Java and C#.
2. Unzip the bundle.
3. Install Erlang using the OTP installer by running otp_win32_R13B03.exe from the unzipped bundle.
4. Create a subdirectory "RabbitMQ" in "C:\Program Files". (Or "C:\Program Files (x86)" if you are running on 64-bit).
5. Unzip the rabbitmq-server-windows-1.8.0 zipfile in the bundle to a directory rabbitmq-server-windows-1.8.0. 
6. Copy the rabbitmq-server-windows-1.8.0 directory you just unzipped to the RabbitMQ directory you just created. Copy the rabbitmq-server-windows-1.8.0 directory itself rather than its contents. ie You should have a directory "C:\Program Files\RabbitMQ\rabbitmq_server-1.8.0" when you have done this.
7. You need to add ERLANG_HOME to the environment variables for your system so RabbitMQ can find Erlang. I can't remember precisely how to do this in XP - I think it's in Control Panel...System... somewhere, and I don't have Windows 7 yet. Perhaps someone could comment for these versions of Windows. In Vista, this is done by:
   1. Open the control panel.
   2. Search "environment". As you type, the option to "Edit the system environment variables" should present itself. Click this - you will be asked for permission to take this action.
   3. In the System Properties...Advanced tab which appears, click the "Environment Variables..." button.
   4. Click the bottom "New..." button to add a system variable.
   5. Enter "ERLANG_HOME" as the variable name.
   6. Enter "C:\Program Files\erl5.7.4" as the variable value for 32-bit systems. Use "C:\Program Files (x86)\erl5.7.4" for 64-bit systems.
   7. Click Ok, Ok, Ok and close the control panel.
8. Run the server by running "C:\Program Files\RabbitMQ\rabbitmq_server-1.8.0\sbin\rabbitmq-server.bat". A command-line window should appear. When it says "broker running" the server is running ok.
9. Install the .Net libraries by running the rabbitmq-dotnet-client-1.8.0.msi installer in the unzipped bundle.

You can also install RabbitMQ as a service. The RabbitMQ documentation has more details on this.

Ok, so now you've installed RabbitMQ, and you have a RabbitMQ broker running (that's the server that all your message producers and message consumers will connect to). Before going any further, it's worth taking a little time to come up to speed with the concepts behind RabbitMQ. Dmitriy Samovskiy has an excellent, and quick, set of introduction slides which should tell you what you need to know (when I clicked publish, I saw that Posterous automatically embedded this - kudos, Posterous crew):

In case you are too impatient to read through that, the gist is that a publisher application sends messages to an exchange, which sits on the server. Message queues sit on the server and filter messages from exchanges. Finally a consumer application consumes messages from a message queue.

A simple publisher

Let's start by writing an application that publishes a message each second which contains "Hello World":

1. Start a new project in Visual C#, called "RabbitMQPub".
2. Add a reference to "RabbitMQ Client Library for .Net", by right-clicking the "References" entry in the Solution Explorer, clicking "Add Reference...", selecting the ".NET" tab, and double-clicking the "RabbitMQ Client Library for .Net" entry.
3. In the program you will want to add using statements to bring in the appropriate RabbitMQ namespaces:
   

   using RabbitMQ.Client;using RabbitMQ.Client.Framing.v0_8;
   </span>

    
4. The first step is to create a ConnectionFactory, and configure it:
   

   ConnectionFactory factory = new ConnectionFactory();
   factory.Protocol = Protocols.FromEnvironment();
   factory.HostName = "localhost";         
   

   
   There are other configuration items, which are taking default values here. This will work on a virgin RabbitMQ instance, but in production you should look at setting up access control and VHosts.
5. Then from the ConnectionFactory, create a Connection object:
   

   IConnection conn = factory.CreateConnection();
   

6. And from the connection, create a Channel object:
   

   IModel ch = conn.CreateModel();
   

7. We need to create an exchange to publish our messages to:
   

   ch.ExchangeDeclare("exch", ExchangeType.Direct);
   

    
8. And a queue to filter messages from the exchange. A queue is made to filter messages from a particular exchange by "binding" it to that exchange:
   

    
   ch.QueueDeclare("queue");
   ch.QueueBind("queue", "exch", "key", false, null);
   

    
9. Then we loop, using the Channel object's BasicPublish method to publish a new message every second, and printing a message to the display to indicate we have done so.
   

    
               while (true)
               {
                   Console.WriteLine("Sending message.");
                   byte[] messageBody = Encoding.UTF8.GetBytes("Hello world");
                   ch.BasicPublish("exch", "key", null, messageBody);
                   Thread.Sleep(1000);
               }
   

    
   Note that we are expected to present byte arrays to send in a message. When sending strings, it's a good idea to use UTF-8 encoding as shown. 
10. Finally, we close the channel and the connection. This will never execute in this case, and the channel and connection will be shutdown by program termination instead, but if you are writing a client that isn't guaranteed to loop forever, this is how you should terminate the connection:
    

     
    ch.Close(Constants.ReplySuccess, "Closing the channel");
    conn.Close(Constants.ReplySuccess, "Closing the connection");
    

     

A simple consumer

We now need to create a simple application to listen for these messages, and print them.

1. Create a new project Follow steps 2-6 and 10 above. This should connect to the RabbitMQ broker, but otherwise is inactive.
2. Add code to consume messages from a queue:
   

   QueueingBasicConsumer consumer = new QueueingBasicConsumer(ch); ch.BasicConsume("queue", null, consumer);
   

3. In an infinite loop, listen for messages, decoding each one, and printing its body
   

               while (true)
               {
                   BasicDeliverEventArgs e = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
                   IBasicProperties props = e.BasicProperties;
                   byte[] body = e.Body;
                   Console.WriteLine("Received message: " + Encoding.UTF8.GetString(body));
                   ch.BasicAck(e.DeliveryTag, false);
               }
   

   
   Note that messages must be acknowledged using BasicAck, or the broker will resend them.
    

Source code

I have made the source code for these projects, as well as other sample projects, through GitHub: http://github.com/edparcell/Samples-CS

Where's the Fudge?

In a previous post, I mentioned that I would like to give an example of using Fudge to encode messages. That is still my intention. However, I thought this blog post had become long enough already, so I'd like to defer that to another post.

Comments

Some random comments on RabbitMQ so far:

• The bundle seems quite large. I know the bulk of this is the OTP installer, but it makes it quite hard to consider embedding RabbitMQ in a lot of small-scale cases, such as deploying to a single end-user machine. Not a problem for an enterprise perhaps, but this is the case I have right now.
  [Update: I'd like to clarify that 80 megs is not a huge download by modern standards, especially for such a useful and full-featured product. Just that it would look odd to my end-users if I replace the named pipes in a few programs with embedded RabbitMQ, and the installer balloons from 1 meg to 80 megs].
• Would be nice to create an installer to automate these steps, so installation is simply clicking "Next" a few times.
• The .Net user guide seems a little out-of-sync with the latest library in places. Not terribly, but it throws you a curveball sometimes, such as suggesting that ConnectionFactory.CreateConnection takes 3 parameters, when in the latest version you should set some member variables on the factory and call the method with no parameters.

Overall, RabbitMQ seems like a great application, albeit a little tough to get started in on .Net. Hopefully this short tutorial will help with that.

I'd gratefully welcome any corrections - as I said, I'm no messaging guru, so if I've met any errors, let me know and I'll correct them.

ZeroMQ looks like an interesting project. It's a light-weight messaging stack, supporting point-to-point, publish-subscribe and other topologies, without a broker.

ZeroMQ aims to sit at a slightly higher level than raw sockets. It's easy to write a socket "Hello World" server and client. The server prints whatever it receives, and the client connects to the server and emits "Hello World". Scaling up to real world needs is hard though. Sockets tend to fall over. Messages certainly will be fragmented. Your socket may be handling several clients at the same time. Messaging layers such as ZeroMQ aim to provide an abstraction, so you can put a string or an object in one end, and have it come out the other with minimal fuss.

In the past, I've looked at ActiveMQ, and RabbitMQ. Both seemed interesting, although as I understand it, both require a broker i.e. a server. This is fine for enterprises, the main target of messaging systems to date, but a little trickier for an individual - sure you can run a broker on your machine, but it's just something to be down at inopportune moments.

As far as I can see ZeroMQ is to Active/RabbitMQ as SQLite is to MySQL or PostgreSQL. You can embed ZeroMQ in your applications, and get the benefit, but the user doesn't have to care, and critically, doesn't have to run a server on their desktop.

And I think that having personal message systems on the desktop is exciting. I already have one, sort of, kind of. Growl is an application that sits in the background listening for "messages". Various applications support growl to pop up status messages. So when a friend comes online, Growl pops up a bubble to notify me. Or when various things update. It's a nice and useful program, but it's not really a messaging system - the "messages" are one-way, and do one thing: pop up on the screen.

A real messaging system goes further. One program could listen for a certain type of message to pass on to Growl. My servers could send messages detailing their status. A third could monitor those messages to check whether limits were breached, and send messages for Growl if so. A library for Excel could monitor messages, and give the latest value for various things, even updating in real-time, allowing quick-and-easy status board. Messages could be sent for stock and other financial data.

People are already doing all of this, of course. But you need to be an enterprise, or prepared to act like one, to use an enterprise messaging system. ZeroMQ will bring the bar down, and in the past, when that's happened, there have always been exciting new products. I hope I've given some, but, as usual, I expect the most exciting new uses will be the ones that we can't even see at the outset.

I've been playing a little with ZeroMQ at weekends, and in a future post, I'll describe how to install ZeroMQ, and give some simple examples of how to use it.

While I'm talking about messaging, I'd like to give a quick mention to the Fudge messaging project who are building libraries to support a slick binary format for encoding messages.

When I have to use a new library or functionality, I tend to build a minimal console application to check that I can get something working in isolation from the codebase it will eventually be integrated into. My first blog post grew out of a sample I was building to try out JQuery and CouchDb. Often, there's good sample code out there on the web, but sometimes not. In any case, I often need to do something different to the other guy, so it's as well to make my own sample.

In the past, I've tended to throw away these samples once they are working, for lack of anywhere reasonable to put them. Recently though, I've noticed that often, great working code samples are available on github, so I've decided to join the crowd there.

My first github repository - Samples-CS - is set of code samples for various things in C#. Currently there are simple console applications to encipher and decipher using AES, convert a PNG to an ICO, and a simple system tray application. They are released under an MIT licence. I'll be adding more as things come up, as well as repositories for other languages.

Hopefully they will be useful to someone who needs to do the same things as me. If not, I lose nothing, and I can refer back to my code samples later if I need to.