Wednesday, 29 July 2009

Interactive "Over-by-Over" Reports using the Guardian Open API

I recently applied for and was given a developer key for the Guardian Open Platform, an initiative from the Guardian newspaper to make available content and data for use by other developers and websites.

My idea was to pull in data from their excellent, and humourous, "over-by-over" coverage of the Ashes cricket, to display on an interactive chart. You would be able to select a particular batsmen and match, view their innings graphically and be able to hover over each over to see the score, state of play and commentary from the time.

And I'm pleased to say it came together very well. Please click to view the result - interactive over-by-over reports - and let me know your thoughts.

Finally, I've also set up a related blog post if you are interested in how it was put together.

Building Interactive "Over-by-Over" Reports

As mentioned in a related post, I've been granted a developer key for the Guardian Open Platform and have used it to put together a website displaying interactive cricket scores and reports. This post details how I've put this together.

Web Service Back End

The application is in two parts. My initial thoughts were to build this purely on the client side using jquery - the API provides JSON as well as XML response formats. But of course this would mean exposing my developer key to anyone who thought to "view source"... and I expected the Guardian wouldn't be too pleased with that. So instead I set up a web service, that will be called by the client side code AJAX style, in order to protect this information.

I also took this opportunity to simplify the output going to the client side code to what I really needed, in order to ease the complexity of the JavaScript manipulations.

The API provides a means of loading the full details of any piece of content via an ID, and a URL of the form:[item]?api_key=[key]

So first step was to find the IDs for each of the matches. I found the easiest way to do this was not to actually use the API explorer, but to find the page on the site, view source and search for
"". The number that comes after this URL (used for the "send to friend" feature), is the ID I wanted.

Having stored these in my web.config file, I built a web service that takes the test match number (1-5), accesses each of the 5 reports (once for each day) and loads them into an XMLTextReader:

    1 string apiKey = ConfigurationManager.AppSettings["APIKey"];

    2 int match = 0;

    3 if (int.TryParse(Request.QueryString["match"], out match))

    4 {

    5     string[] contentItems = GetContentItemsForMatch(match);

    6     foreach (string contentItem in contentItems)

    7     {

    8         string url = "" + contentItem + "?api_key=" + apiKey;

    9         XmlTextReader reader = new XmlTextReader(url);

   10         ...

As it turns out, although the content is well tagged as XML, the body copy all sits within a single node (though paragraphs are marked up). So I extracted the node I needed:

    1 if (reader.NodeType.ToString() == "Element" && reader.Name == "body")

    2     body = reader.ReadElementContentAsString();

And then carried out some string manipulations to tidy up the content and extract it into a JSON collection of the form:
{overs: [
day : 1,
innings : 1,
over : 1,
score : "England 2-0 (Strauss 1, Cook 1)",
batsmen: [
{name : "Strauss", score : 1},
{name : "Cook", score : 1}
text : "It'll be Mitchell Johnson to bowl The First Ball..."
I discovered the API seems to have some form of restriction on frequency of requests, as I would sometimes find my request for day 4 or 5 would fail with a "403 forbidden" message. To get around this I added some error trapping via try/catch/wait/retry, and also some caching to prevent hitting the Guardian services too often.

Finally, having built up the JSON string using a StringBuilder, I output it to the response stream:

    1 Response.ContentType = "text/js";

    2 Response.Write(sb.ToString());

Client Side Code

The front-end of the application consists of a single page, with interaction with the web service achieved using JavaScript, and in particular, jquery.

The core of this is a function wired up to the button click that retrieves the JSON formatted data for the selected match, and parses it to pull out the over objects that correspond to the selected batsman's innings.
function setUpButtonClick() {
$("#show-button").click(function() {
var ddl = $("#match")[0];
var match = ddl.options[ddl.selectedIndex].value; //get selected match
$("#wait-icon").show(); //show ajax "please wait" graphic
$.getJSON("/interactive-obos/services/get-overs/?match=" + match, function(json) {
hideOverDetails(); ///hides any previously selected over information
displayInningsDetails(json.overs); //plots the innings details on the chart
The function displayInningsDetails() takes the JSON collection over over information and plots it on the chart. This is just a div with a grid background image, and absolutely positioned elements are placed on top of it using jquery append statements.
function displayInningsDetails(overs) {
var ddl;
ddl = $("#batsman")[0];
var batsman = ddl.options[ddl.selectedIndex].value;
ddl = $("#innings")[0];
var innings = parseInt(ddl.options[ddl.selectedIndex].value);
var inningsOver = 0;
for (var i = 0; i < overs.length; i++) {
var j = getBatsmanInOver(overs[i],batsman,innings);
if (j > 0) {
$("#display-panel").append("<div class=\"over\" title=\"Runs: " + overs[i].batsmen[j-1].score + "\" style=\"left: " + ((inningsOver - 1) * OVER_WIDTH) + "px; height: " + ((overs[i].batsmen[j-1].score) * RUN_HEIGHT) + "px\"></div><div class=\"text\"><strong>Day " + overs[i].day + "</strong><strong>" + overs[i].score + "</strong>" + overs[i].text + "</div>");

function getBatsmanInOver(over,player,innings) {
if (over.innings == innings && over.batsmen[0].name.toLowerCase() == player.toLowerCase())
return 1;
if (over.innings == innings && over.batsmen[1].name.toLowerCase() == player.toLowerCase())
return 2;
return 0;

And finally, the hover function to display the over information is set up. The text has already been written into a hidden div within the chart, so the steps are to find the one next to the visible bar, extract the HTML and write it to the panel.

function applyOverHovers() {
$("#display-panel div.over").hover(function() {
$("#text-panel div").html($(this).next("div.text").html()).animate({opacity: "show"}, "slow");
}, function() {
$(this).next("div.text").animate({opacity: "hide"}, "fast");

And that's broadly it! If you'd like to dig-in further, please feel free to download the source code and of course let me know your comments.

Monday, 20 July 2009

Setting Up Ruby on Rails on Vista

Possibly going slightly around the houses, but I'm sure similar to other developers who have started using ASP.Net MVC, I've recently started setting up and taking a look into Ruby On Rails. Only had an quick look so far, and had a read through the rails guides to get started - but already can see why it's proved such a popular environment for web development, how it's influenced the development of ASP.Net MVC, and how it's come to promote the MVC pattern more widely in web development circles.

Also probably oddly for a RoR developer, but similar to others from a Microsoft background, I'm running Vista. There are some useful posts out there for setting up RoR on Vista, but as I found a few issues that weren't covered figured it worth noting here.

MySQL Setup

Whilst setting up Rails I also installed MySQL to use as the database. The link provided above provides all the details for installing this. Note though that when it comes to configuring the installation by running MySQLInstanceConfig.exe make sure to right-click and use the "Run as administrator" option - otherwise the latter parts of the wizard won't be able to complete.

There's also a useful recommended step to make the installation more secure by restricting access to the local machine. This is done by adding a line bind-address= to the [mysqld] section of my.ini. Note though that you first need to change the security settings to allow modification of this file (right-click on file, go to Properties, go to Security tab, select Edit and click the Modify option).

RoR With MySQL

Finally had a couple of issues with running a simple RoR application running MySQL. This firstly manifested itself with an error reporting that "libmysql.dll was not found". Copying this file from the MySQL install directory to ruby\bin seemed to fix that.

But still the setup wasn't very stable. Every few time I ran the application the WebBrick web server would crash with Ruby interpreter (CUI) 1.8.6 [i386-mswin32] has stopped working. This seemed to be only the case with the latest version of MySQL though. Removing this and installing the previous version (5.0) seemed to resolve this though, as others have found.

Monday, 6 July 2009

Serialization and Deserialization of Class with Changed Structure

In building a CMS versioning system, I had the idea of using database saved, serialised versions of my objects to represent revisions - which could then be loaded and reverted to the live record in the database when required.

A potential problem though was what if my object structure had changed between the save and the retrieval? i.e. what if I add a new field to my object, that won't be in the serialised data.

I wrote this test to find out:

    1 using System;

    2 using System.Collections.Generic;

    3 using System.Linq;

    4 using System.Text;

    5 using System.IO;

    6 using System.Runtime.Serialization.Formatters.Binary;


    8 namespace SerializationTests

    9 {

   10     [Serializable]

   11     class SampleClass

   12     {

   13         private int mintID;

   14         private string mstrName;

   15         private string mstrName2 = "Default value";

   16         private int mintNumber = 2;


   18         public int ID

   19         {

   20             get { return mintID; }

   21             set { mintID = value; }

   22         }


   24         public string Name

   25         {

   26             get { return mstrName; }

   27             set { mstrName = value; }

   28         }


   30         public string Name2

   31         {

   32             get { return mstrName2; }

   33             set { mstrName2 = value; }

   34         }


   36         public int Number

   37         {

   38             get { return mintNumber; }

   39             set { mintNumber = value; }

   40         }

   41     }


   43     class Program

   44     {

   45         static void Main(string[] args)

   46         {

   47             //SerializeToFile();

   48             DeserializeFromFile();

   49             Console.ReadLine();

   50         }


   52         static void SerializeToFile()

   53         {

   54             SampleClass obj = new SampleClass();

   55             obj.ID = 1;

   56             obj.Name = "Test";

   57             using (FileStream stream = new FileStream("c:\\temp\\object", FileMode.Create))

   58             {

   59                 BinaryFormatter formatter = new BinaryFormatter();

   60                 formatter.Serialize(stream, obj);

   61                 stream.Close();

   62             }

   63             Console.WriteLine("Serialized object to file...");           

   64         }


   66         static void DeserializeFromFile()

   67         {

   68             SampleClass obj;

   69             using (FileStream stream = new FileStream("c:\\temp\\object", FileMode.Open))

   70             {

   71                 BinaryFormatter formatter = new BinaryFormatter();

   72                 obj = (SampleClass)formatter.Deserialize(stream);

   73             }

   74             Console.WriteLine("Deserialized object from file...");

   75             Console.WriteLine("ID: " + obj.ID);

   76             Console.WriteLine("Name: " + obj.Name);

   77             Console.WriteLine("Name2: " + obj.Name2);

   78             Console.WriteLine("Number: " + obj.Number);      

   79         }

   80     }

   81 }

Which gave me the following result:

ID: 1
Name: Test
Number: 0

Upshot seems to be that for new fields, the deserialisation works without errors - i.e. it doesn't fail when it can't populate a field from the serialised data. But the new fields take on the value of the default for the data type (e.g. 0 for int, empty string for string) rather than the default value set in the private member variable.