This page has moved

My blog has moved to http://www.antonydenyer.co.uk

HOWTO: Deploy mono web applications with capistrano-mono-deploy

I've recently been working on a capistrano gem to help with mono deployments. I've had a bit of experience trying to get capistrano to work with non-rails environments, basically it can be a bit of pain if you're not sure what you're doing. Also most of the beginners tutorials are aimed at people developing Ruby on Rails. So I though I'd create a gem and provide a little how to. The gem is called capistrano-mono-deploy. This is how you can use it to deploy a simple web application to a Linux machine. In this example I'll be using the excellent ServiceStack for the web application and xsp4 to host the it. You can find all the code on my github account.

Mono

I'm running Ubuntu 12.10 on my server and am using Mono 2.10.8.1 which was installed from the package repository using apt-get

sudo apt-get install mono-xsp4

Service Stack

First I knocked a quick service stack application and created a status endpoint. This should work for any application that's mono compatible but I haven't tried with anything else. Once you've got an application you'll want to be deploying it.

Capistrano

You need to make sure you have capistrano and ruby gems installed first. I'm running this from my Ubuntu development machine and have not tried deploying from a windows machine to 'nix machine.

Install ruby and ruby gems

sudo apt-get install ruby1.9.3 rubygems

Install bundle

sudo gem install bundle capistrano

Add a Gemfile and insert the following

source :rubygems
gem "capistrano-mono-deploy"

Call bundle to update the gems

bundle

Capify your project

capify .

Edit your config/deploy.rb to be something like this:

require "capistrano/mono-deploy"

ssh_options[:keys] = %w('~/.ssh/*.pub')

set :application, "service stack deployed with capistrano running on mono"
set :deploy_to, "~/www" 

role :app, "the.server.com"

Then from the root of your project call

cap deploy

BOOM your application is deployed!

Assuming you have your ssh keys setup for the server, either that or you had to enter a password. 

Now I wouldn't recommend using xsp as your weapon of choice when it comes to hosting, but it's easy to get started with and you can swap it out for other things once you get up and running.

You should take some time to read up about capistrano and how it works. It's a really powerful tool that has loads of options available for you to play with. The capistrano-mono-deploy gem currently deploys the first directory it finds with a web.config located in it.

Feedback appreciated, particularly on my terrible ruby code!

Deploying ServiceStack to debian using capistrano with hosting in nginx and fastcgi

Over the last month we've started using ServiceStack for a couple of our api endpoints. We're hosting these projects on a debian squeeze vm using nginx and mono. We ran into various problems along the way. Here's a breakdown of what we found and how we solved the issues we ran into. Hopefully you'll find this useful.

Mono

We're using version 2.11.3. There are various bug fixes compared to the version that comes with squeeze. Namely, problems with min pool size specified in a connection string. Rule of thumb, if there's a bug in mono then get the latest stable!

Nginx

We're using nginx and fastcgi to host the application. This has made life easier for automated deployments as we can just specify the socket file based on the incoming host header. There's a discussion about what's the best way to host ServiceStack on Linux at stackoverflow. Our nginx config looks like this:
When we deploy the application we nohup the monoservice from capistrano Where: fqdn is the fully qualified domain name, i.e. the url you are requesting from latest_release is the directory where the web.config is located.

Capistrano

To get the files onto the server using capistrano we followed the standard deploy recipe. In our set up we differ from the norm in that we have a separate deployment repository form the application. This allows us re-deploy without re-building the whole application. Currently we use Teamcity to package up the application. We then unzip the packaged application into a directory inside the deployment repository.
We us deploy_via copy to get everything on the server. This means you need to have some authorized_keys setup on your server. We store our ssh keys in the git repository and pull all the keys into capistrano like this:

ssh_options[:keys] = Dir["./ssh/*id_rsa"]

No downtime during deployments ... almost

Most people deal with deployment downtime using a load balancer. Something like take server out of rotation, update the server, bring it back in. The problem with this is it's slow and you need to wait for server to finish what it's doing. Also in our application we can have long running background tasks (up to 1000ms). So we didn't want to kill those of when we deployed. So we decided to take advantage of a rather nice feature of using sockets. When you start fastcgi using a socket it gets a handle to the file. This means that you can move the file! Meaning that you can have a long running process carry on whilst you move you new site into production, leaving that long running task running on the old code to finish. Amazing!
This is what we have i so far:
There's room for improvement feedback appreciated.

ServiceStack the way I like it

Over the last month we've started using ServiceStack for a couple of our api endpoints. Here's a breakdown of how I configured ServiceStack to work the way I like it. Hopefully you'll find this useful.

Overriding the defaults

Some of the defaults for ServiceStack are in my opinion not well suited to writing an api. This is probably down to the frameworks desire to be a complete web framework. Here's our current default implementation of AppHost: For me, the biggest annoyance was trying to find the DefaultContentType setting. I found some of the settings unintuitive to find, but it's not like you have to do it very often!

Timing requests with StatsD

As you can see, we've added a StatsD feature which was very easy to add. It basically times how long each request took and logs it to statsD. Here's how we did it:
It would have been nicer if we could wrap the request handler but that kind of pipeline is foreign to the framework and as such you need to subscribe to the begin and end messages. There's probably a better way of recording the time spent but hey ho it works for us.

RestServiceBase, Exceptions and Error Responses

One of my biggest bugbears with ServiceStack was the insistence on a separate request and response object, the presence of a special property and that you follow a naming convention all in the name of sending error and validation messages back to the client. It's explained at length on the wiki and Demis was good enough to answer our question.

RestServiceBase

The simple RestServiceBase that comes with provides an easy way of getting started, but there aren't many hooks you can use to manipulate how it works. It would be nice if you could inject your own error response creator. We ended up inheriting from RestServiceBase and overriding how it works:
We basically chopped out the bits we're not using and changed the thing that creates the Error Response: This allows us to respond with an error no matter what the type is for the request or what response you are going to send back. It provides us with extra flexibility above what is provided out of the box. In a nutshell, if there's an exception in the code we will always get a stack trace in the response if debug is on.

Validation

We had the same issue with the validation feature; if you don't follow the convention you don't get anything in the response body. So we followed the same practice and copied the ValidationFeature and tweaked it how we wanted it.

Conclusion

I like ServiceStack; it's really easy to get up and running and whilst it has it's own opinions on how you should work, what framework doesn't?

Run All sql files in a directory

Create a a batch file and change the params accordingly to run all sql files in a directory against your db

Stub responses with Nest.ElasticClient

We've started using ElasticSearch at work for some of our projects. When we started out doing simple web requests was easy enough but as the complexity of what we where doing grew it became obvious that we where starting to write our own DSL for elastic search. Surely someone else has already done this?

There are several options but we settled on NEST. Mainly because of the way the DSL was written. The first stumbling block I hit was unit testing. You could just stub out IElasticClient and then add some integration tests to cover yourself.

The problem with integration tests is they're slow and require, well something to integrate with, so I thought it would be good to stub the json response from ElasticSearch.

When using NEST it's not all that obvious. But here's how I did it:


The down side is you're testing the frameworks assumptions. But it's good to know what the framework is actually doing under the hood i.e. calling a web request with some parameters.

You also get the added bonus of being able to test that the response can be parsed into you're objects.

Doing to much in one test?

Absolutely, in our code base it's two tests. One for parsing the response and another test to assert post parameters.

Cache Fluffer / Gorilla Caching / Cache Warmer

The relatively simple introduction of a cache fluffer can make a huge difference to performance, particularly at peak load. The idea is simple, keep the cache up to do date so you don't have to go and get data when the user requests the site.

        public T Get(Func<T> factory, string cacheKey)
        {
            if (cache.Contains(cacheKey))
            {
                Task.Factory.StartNew(() => FluffTheCache(factory, cacheKey));
                return cache.Get(cacheKey);
            }
            return Put(factory, cacheKey);
        }

        public void FluffTheCache(Func<T> factory, string cacheKey)
        {
            var expiry = cache.getExpiry(cacheKey);
            if (expiry.Subtract(new TimeSpan(0, 0, 10)).Second < new Random().Next(1, 60))
            {
                Put(factory, cacheKey);
            }
        }

        public T Put(Func<T> factory, string cacheKey)
        {
            var item = factory();
            cache.Set(item, cacheKey);
            return item;
        }

The beauty with this method is that popular items will always be in cache where as less popular items will not be cached for unnecessary lengths of time.

You also get some randomization mixed in for free meaning that items don't come out of sync at the same time.

Testing Async Behaviour - AutoResetEvent WaitOne

This little nugget helps you test async code.

https://github.com/7digital/SevenDigital.Api.Wrapper/blob/master/src/SevenDigital.Api.Wrapper.Unit.Tests/FluentAPITests.cs

Essentially you use a semaphore to signal that something has finished. In our case we're using the AutoResetEvent. This allows to wait up to an amount of time for something happen.

In our case we're waiting for an action to be fired and we want to ensure the result is true.

[Test]
public void AsyncTest()
{
  var autoResetEvent = new AutoResetEvent(false);
  new Something.Async(
    action =>
    {
 Assert.That(action.SomethingToTest, Is.True);
 autoResetEvent.Set();
    });

  var result = autoResetEvent.WaitOne(1000 * 60);
  Assert.That(result, Is.True, "Method Not Fired");
}

Make sure that you test the result from WaitOne. If you don't you're not testing that the method actually returns.

Samsung Chromebox with XBMC

What to do with a Chromebox?

At google IO 2012 every delegate got a free Samsung Chromebox. Personally I already have a laptop, desktop, tablet and smart phone. Why would I need a cut down desktop? It would probably be great for your Nan who has no idea what they're doing and just wants to email the grandkids.

So what should I do with this piece of hardware? Well my xbox classic is struggling to play some high def media files, it is over 10 years old, how about I use my free chromebox.

Enable Developer Mode

Essentially flip a switch and erase all user data. Here's how.

Install ChrUbuntu

I only had one problem following these instructions, me, I didn't follow them. You must leave your box in developer mode, meaning that boot time takes almost a minute because of the developer mode warning.

Install XBMC

XBMC has been accepted into debian.

sudo apt-get install xbmc

BOOM!

Install VNC and SSH

Invaluable, otherwise you'll be plugging in your keyboard and mouse every time you get a problem. 

Install SSH
Enable Remote Desktop Connections

Run xbmc on startup

I had a slight issue with xbmc starting just before the window manager had fully started. This was causing xbmc to start in a windowed state. Instead I wrote a small script to pause for 2 seconds then start xbmc.

#!/bin/sh
sleep 5
xbmc

chmod +x and then add the file into the script to startup programs.

Summary

In total it took me about 90 minutes. My main annoyance is the beep you get on startup. I connected it to my TV through the dvi port. It handled 720p files with no problems, just need to test with some 1080p files.

Just waiting on my display port -> HDMI cable  so that I can give it a proper testing.

Would I recommend buying one explicitly for this purpose, no, unless the price tag is brought down significantly.

Language Obsessed

Recently on the GOOS group discussion board there was a good debate about Acceptance Testing. Whilst much of the advice was good it amazed me how many people hadn't looked outside of their own language community for inspiration.

For instance, in the dot net world automated acceptance testing frameworks for web sites where fairly poor. However the Ruby guys nailed this a while ago so why not just use those frameworks? Take a look at WATIR for more information. When it comes to black box testing your application why does it matter what language or framework you use? If anything I'd say it's better to test it using a completely different language.

If you've written your application in C# what's stopping you from using Ruby to test you website? Many people get scared of using something that isn't written in their favorite language. The problem with this is you don't expose yourself to new ideas a new ways of doing things.

Don't limit yourself to one ecosystem.

Cucumber.js with zombie.js

I wanted to start looking at alternatives to our current set of cucumber feature tests. At the moment on the web team we're using using FireWatir and Capybara. So I though I'd take at look at what was available in Node.js. Many people think it's strange that a .Net shop would use a something written for testing Ruby or even consider something that isn't from the .Net community. Personally I think it's a benefit to truly look at something form the outside in.  Should it matter what you're using to drive your end product or what language your using to test it? Not really. So what are the motivations for moving away from Ruby, Capybara and FireWatir?

In a word 'flaky', we've had heaps of issues getting our feature tests, AATs and smoke tests reliable. When it comes to testing, consistency should be king. They should be as solid as your unit tests.  If they fail you want to know that for definite you've broken something, rather than thinking it's a problem with the webdriver.

It is with this aim in mind that I started looking at the following.

Cucumber.js is definitely in it's infancy, there's lots of stuff missing but there's enough there to get going.

Zombie.js is a headless browser, it claims to be insanely fast, no complaints here.

First up we got something working with the current implementation of cucumber-js https://github.com/antonydenyer/zombiejsplayground. The progress formatter works fine and the usual "you can implement step definitions for undefined steps" are a real help. Interestingly rather than requiring zombie.js in our step definitions we ended up going down the route of implementing our own DSL inside world.js. We could have used another DSL like capybara to protect us from changing the browser/driver we use. This is currently done with our Ruby implementation, the problem is that we've ending up implementing our own hacks to get round the limitations/flakiness of selnium/webdriver and to date we have never 'just swapped out the driver' to see what happens when they run against chrome/ie. That said should you be using cucumber tests to test the browser? I don't think you should. With that in mind we ended up implementing directly against zombie.js from our own DSL.

Extending cucmber-js https://github.com/antonydenyer/cucumber-js

There are a lot things yet to be implemented in cucmber.js one that gives me great satisfaction is the pretty formatter. Look everything is green!  It's no where near ready for production but you do get a nice pretty formatter.

Thanks to Raoul Millais for helping out with command line parsing and general hand holding around JavaScript first steps.

OpenRasta is opening up to the community

Last Thursday a few of us from 7digital had a meet up with Sebastien Lambla author of OpenRasta.

As some of you may know we're been writing all our new API endpoint using OpenRasta, we have a vested interested in ensuring the success of this project and as such are responding to the rallying cry with gusto.

So what's going to happen? Essentially 7digital, along with Huddle and Neil Mosafi, will jumping on board to help out with development and maintenance of OpenRasta 2.x. Short term goals are to help people get started with OpenRasta. At the moment it's not particularly easy to get going with the 2.1 code. The first thing to get up and running with is a build server, this is something that 7digital will be taking responsibility for. Our first aim is to build OpenRasta and publish _latest binaries with every push and make those binaries available in OpenWrap, NuGet and as binary downloads.

We're really looking forward to working with everyone on OpenRasta and can't wait to get stuck in.

Out of band caching

One of the things we try and do is always keep our site up, sounds simple right? One way to achieve this is to have a good caching policy.

Here's what we do:

• Cache a response from some external service (be it a db or web service, whatever)
• When a user request the resource ALWAYS serve it from cache if it's there
• Then "out of band" (i.e. on a different thread) make a call to get it from the external service and update the cache if you get a valid response.
• But only do this if the cache is stale

Simple right? That way if your external service goes down you can still serve it from cache. Your user will always get the fastest possible response (i.e. they wont be the sucker who has to go and get it). And your service remains up as much as possible.

Double Entry Accounting and TDD

Double-entry bookkeeping system

A double-entry bookkeeping system is a set of rules for recording financial information in a financial accounting system in which every transaction or event changes at least two different nominal ledger accounts.

At it's simplest you have two ledgers when you make an account transaction you make an entry in both ledgers. Then at the end of the month you reconcile these two ledgers and they should be the same. Essentially from an accounting point of view we're saying that by using two different ways of doing something we come to the same answer.

Why is the useful from a TDD perspective?

Well you should be using different ways to make your assertions from your implementation, particularly when you talk about integration tests.

Lets say your writing some integration tests for a repository. Lets say you're using Simple.Data to access your database. Your tests should then use something else. You should probably get as close to the metal possible. In this case you should probably use the SqlCommand class.

Put some data in your database using SqlCommand. Then assert that you can retrieve it using Simple.Data.

That way you are making a Double Entry Assertion in your tests. The chances of both systems being broken in the same way are reduced.

Red feature tests are pointless

We've spent a lot of time recently on fixing up our automated feature tests (AATs). The problem has been that these failing tests have blinded us to real problems that have crept into live systems. The usual answer is to 'just run them again' and eventually they go green. The main problem is our attitude to the tests, we don't respect them and we don't listen to them, as such they provide no feedback and are completely pointless. The response to broken feature tests would normally range from the test environment is down, the data is in contention, the database is down etc etc, but never something I've done has broken something.

So what is the solution?

We improved the reliability of the tests that we could and removed the ones we couldn't. Now you may think this is a bit of a cop out, but the amount of feedback we were getting from the tests was negligible. The best think you can get from your tests is something you don't expect. you should expect them to pass and be surprised or shocked when they don't.

Stop the line and fix the problem. Don't just keep running them again.

MVC and our interpretation at 7digital

Introduction

The following details concepts that have been adopted on the site I'm currently working on. This was born out of a discussion that took place among the devteam at the time. Differences in phraseology and meaning were ironed out to come up the following definitions and responsibilities.

Queries(GET)/Commands(POST)

* Contain all the information required to perform an action.
* Data should be bound onto the query before it hits the controller action.
* They are POCO and contain no logic.

Controllers

* Marshall request to handlers and mappers
* Creates the view model from the handler result

Handler

* Process queries/commands
* returns various types (Model/Entity/API DTO)

View Models

* Contains data that you are using in the view, should not expose anything other than view models/value types.

View

* Stuff you show the user, should only contain logic that is required to get the stuff displaying correctly.

Wrappers

* Wrap calls to external api

Services

* Do things that aren't simple, pull together bits and bobs and package them up. Take care of business logic that have cross cutting concerns.

Mappers

* Mappers simply convert one object to another. They DO NOT perform any other logic.

Charts Example from http://www.7digital.com/b/charts

ArtistChartController

* Takes the query from the web request (which has been bound in the pipeline)
* Passes the query to a handler
* Maps response from handler to view model using auto mapper

ArtistChartQueryHandler

* Takes the query
* Does some work to get some configuration options
* Builds up the query further to execute against the API wrapper
* Returns API DTO

Api Wrapper

* Takes query
* Executes against external API
* Parses responses and maps into API DTO object

Resolving an open generic type with Castle Windsor

One of things I wanted to do the other day was resolve an open generic interface. Or more specifically resolve a generic type at runtime.

This is what we came up with:

var argumentsAsAnonymousType = 
 typeof(IHandler<>)
  .MakeGenericType(instance.GetType());

var concrete = 
  IoC.Container.Resolve(argumentsAsAnonymousType);

The first problem was resolving an open generic type, this was solved using MakeGeneric. This returns a type object which you can pass to windsor to resolve for you.

But now you're left in a situation where you don't know what the returning type is. You could use reflection to call the method.  The good thing about this is in the type you're declaring you can afford to use a generic type.

public interface IHandler<T>
{
    void Handle(T instance);
}

public class FileNotFoundHandler : IHandler<FileNotFoundException>
{
    public void Handle(FileNotFoundException exception)
    {
        _logger.Warn(fileNotFoundException.FileName)
    }
}

The downside is, you're using reflection! The alternative is to use a non generic interface and utilize interface inheritance.

public interface IHandler<T> : IHandler
{

}

public interface IHandler
{
   void Handle(object instance);
}

public class FileNotFoundHandler : IHandler<FileNotFoundException>
{
    public void Handle(object exception)
    {
        var fileNotFoundException as FileNotFoundException;
        _logger.Warn(fileNotFoundException.FileName)
    }
}

That way when you resolve the handler you can cast it.

var argumentsAsAnonymousType = 
 typeof(IHandler<>)
  .MakeGenericType(instance.GetType());

var concrete = 
  IoC.Container.Resolve(argumentsAsAnonymousType) as IHandler;

concrete.Handle(exception)

The benefit now is that you don't need to use reflection. The down side is that you need to unbox your object inside the handler, however this shouldn't be an issue as you've already guaranteed the type when you resolve the object.

Take It To Systest

One of the things I want to try to change at my current work place is to increase the level of developer responsibility when it comes to acceptance testing.  What hapens at the moment is a pair of develoers will pick up a card, work on it by writing unit tests and acceptance test then at some point they will decide it's 'done'.  At this point it will be 'pushed' to systest for testing by QA.  Inevitably some issues will be found and the work will be handed back to development, where it will wait to be picked back up again.

What I would like see is one of the two developers working on the card to take ownership of the acceptance testing and to 'take it to systest' to be tested with the help of the QA team.  That way small bugs that are always found in testing can be sorted out straight away.  The development would eventually get a better understanding on how other people use their system, defects found in QA are ironed more quickly, thus improving project velocity.

PostSharp Collaborators

One of the things we've been working on recently is caching with Memcache.  Caching is a cross-cutting concern that we thought could be refactored into an aspect.  Thus making caching happen automatically with the small addition of an attribute.  We went down, what seemed, the obvious route of postsharp and duly made and aspect that took care of our caching needs.

The problem we had with our aspect was getting the instance of the cache into the aspect. Normally we would include this in the constructor as a collaborator.  Obviously we can't do that because the aspect needs an empty constructor.  So what are the options?

1. Pass the instance into the method where the aspect is being used, then pull it out in the aspect by getting the method args. 
2. Use a service locator/factory or some other mechanism to get hold of the instance.

We tried the first option and it wasn't particularly satisfactory, it looks like your passing something into the method that you're not using in that method call (exasperated by R# telling you you don't need it). Not ideal! So we went down the line of getting hold of the instance using a service locator.

Since we went live with I found out that Castle Windsor has some limited options for AOP, it's called an IoC Interceptor this gives you the ability to intercept every method call on an interface.  This is fine for what we have as all the interface calls require caching.