Thursday, 3 January 2019

Maintaining the Sitecore Helix Architecture

Like most development teams working on modern, Sitecore projects of reasonable complexity, at the agency where I work we've followed the Helix design principles and conventions for Sitecore development in structuring the solution. This means we have 3 groups of projects:

  • Foundation - at the lowest level, these projects may depend on each other but don't depend on any in the higher levels. Modules in the Foundation layer are business-logic specific extensions on the technology frameworks used in the implementation - e.g. Sitecore itself - or shared functionality between feature modules that is abstracted out.
  • Feature - these are projects in the middle layer, which may depend on Foundation projects but not on any in the higher Project level and, importantly, not on each other. Projects here should map to a business domain concept or feature.
    • We've extended the feature level in a small way by introducing the concept of sub-features. For example, for a data import feature, we have an API project, processing implementations (using Azure Function and console apps), and a common logic project. These are all classed as a single feature and dependencies within sub-features are allowed.
  • Project - this is the highest level and can depend on any of the lower levels. It's where the website output is pulled together by combining features, into a web application output. We have a common one, and then one for specific details of each of the sub-sites in our multi-site solution.

Ensuring Adherence to the Helix Architecture

There are two forms of dependencies that go against the Helix principles and rules defined above. Hard dependencies are in the form of project references - if one feature project directly depends on another one for example. In addition to developer guidance and code reviews, we have a defence against this in the form of an architectural fitness function that runs are part of the unit test suite - thus failing a build if the conventions are broken.

The other form of dependency are soft ones, that can't be detected by automated tools so need to be part of developer consideration when implementing features and checked in code reviews. These take the form of one project having "knowledge" of another one that should be off limits, perhaps via a shared name of a template or field.

If there seems to be a need to break the Helix conventions, there are a number of ways we've used to resolve it, that can be summarised into the following three methods.

Selecting the Seams

One of the challenges when selecting which projects we should have in which layers is defining the appropriate seams between different groups of functionality. The idea of a feature project in particular, is that it follows the common closure principle which states that "classes that change together are packaged together". It's quite possible that we don't get that right first time, and as the solution develops we look to refactor to either combine or split feature projects. We need to strike a balance here between having useful separation so we follow a form of single responsibility principle for a feature, but not split too much and then find we have a lot of leakage of knowledge between features.

Introducing a Foundation Project

A standard way of resolving the conflict when two feature projects need to share logic is to extract that logic into a foundation project. For example, low-level, cross-cutting concerns such as search indexing and ORM usage are defined in specific foundation projects that are referenced and shared by multiple feature ones. As we move forward, we'll likely want to introduce more of these.

Foundation projects can depend on each other, so we don't have the same concerns at this level of the architecture. However we are of course restricted in that we can't have circular references, so there still may need to be further refactoring and splitting of logic here, such that the foundation projects can depend on each other as needed.

When it comes to softer dependencies - such as knowledge of how different sites (defined at project level) or templates (defined at feature level) behave, typically here we've introduced a foundation level "register" class. These each consist of a static collection of typed objects that each higher level project can register details of at application start-up. Other projects can then reference these registers to extract the details they need for particular renderings, computed fields or other operations.

One example of this is in a class we've called AddressableTemplatesRegister - this maintains a global dictionary of registered template names for those pages that are addressable (i.e. have URLs), such that the indexing routines can ensure to index a URL for those items. The key for the dictionary is the template namd and the value is an instance of a class which has properties for related details such as canonical URL generation, page title generation, and retrieval of site map entries.

Using these each feature project can register at start-up the set of templates that it's concerned with, along with the specifics of how concerns such as those methods are defined. Individual feature projects - e.g. we have a "Navigation" one that amongst other things can enumerate the register and retrieve site map entries appropriate for each template, perhaps filtering out certain ones based on the contents of particular fields.

Delegating to Feature Projects

Whilst Sitecore is particularly extensible via it's pipeline architecture, some concepts only exist as singletons and as such can't be "decorated" by individual features. These are rare - most things, like for example item resolution, can be defined in the appropriate features, and there can be several, each handling different templates. But one example is the LinkProvider - there can only be one of these defined for the solution.

Whilst we have only one of these, defined in the common "Project" project, we delegate the specific details of link generation for different types of pages to the appropriate feature projects, making calls to methods defined on classes there.

Wrapping Up

Using a combination of these methods we've been able to maintain a solution as it's grown over 18 months to continue to be in adherence to the Helix conventions.

Monday, 19 November 2018

Azure Table Storage Batch Insert Within Operations Limit

When working with Azure Table Storage and needing to bulk insert a number of records, it's more efficient and reliable to do so by inserting a batch of records in one step, rather than one at a time.

On a recent project we've been using table storage as a location to record the log for an import operation migrating information from one content management system into another. As each field was imported a couple of log lines of information were recorded - which were useful for auditing and debugging.

Rather than inserting each log line as it was generated, we've batched them up, and then loaded them into table storage in the finally part of a try/catch that sits around the import logic. In this way, we store all the log information in one go.

For a few imports though, we found this error being recorded in the application logs:

Error processing message, message: An error has occurred. Exception message: The maximum number of operations allowed in one batch has been exceeded. Type: System.InvalidOperationException

What I hadn't realised was that the Azure ExecuteBatch and ExecuteBatchAsync statements have a limit of 100 records that can be inserted at once.  Go over this, and you get this error.

My solution to this was to create a new extension method, that "batches up the batch", ensuring that we apply the updates in several batches, each one at or less than the limit.

The extension method looks like this:

And the helper extension method to group the original batch into chunks of the appropriate size is as follows:

Saturday, 10 November 2018

Pluralsight Course: Azure Custom Vision Service

Over the past few months I've been working on authoring a first course for Pluralsight on the topic of the Custom Vision Service, part of the Cognitive Services suite provided by Microsoft and hosted on Azure.

It was released a couple of days ago, and those with subscriptions or a free trial, can access it here:

Saturday, 1 September 2018

Streaming mp3s from Azure blob storage gotcha

Have been wrestling with an issue one and off for a few weeks where have found mp3 files that previously played correctly in a web browser were having some issues relating to streaming. There's an example here. What I found - hopefully resolved now if you try the link - was that the audio would play, but the play bar that updates the position on the track wouldn't move, nor could the user navigate to parts of the audio to review it.

We're using jplayer and have the mp3 files hosted in Azure blob storage. All had worked fine for several years, but recently could see this issue on Chrome, even though they still played as expected in Edge. So must have been related to a relatively recent Chrome update.

The resolution though turned out to be a setting on the blob storage account, that I was led to via this Stackoverflow answer, that indicated setting the DefaultServiceVersion to an appropriate value might resolve it. And sure enough, when querying the current value I found it was null. Setting it to 2013-08-15 (and perhaps more recent versions) restored the play bar and streaming behaviour.

Refactoring an interface to facilitate unit testing

Short post to follow, generalising something I've just done in an active project that seems to have general relevance. I had an interface and class like this, that wasn't easy to put under a unit test.

interface ISomething
{
    string GetValue();  
    string GetComplicatedConvertedValue();
}

class MySomething : ISomething
{
    string GetValue()
    {
        // Get value from something that's a simple one-liner call but not easy to mock or stub (in my case, Sitecore settings)
        ...
    }
    
    string GetComplicatedConvertedValue()
    {
        var rawValue = GetValue();
        
        // Do some processing on the raw value that ideally we'd like under unit test, and return it
        ...
    }
}

One answer I realised is that, as the main value for testing this class is in the second method - retrieving the raw value is hard to test, but little value as it's a one liner into platform functionality - I can just remove that from the interface, and instead implement it as a testable extension method.

interface ISomething
{
    string GetValue();
}

class MySomething : ISomething
{
    string GetValue()
    {
        // Get value from something that's a simple one-liner call but not easy to mock or stub (in my case, Sitecore settings)
        ...
    }
}

static class SomethingExtensions
{
    static string GetComplicatedConvertedValue(this ISomething something)
    {
        var rawValue = something.GetValue();
        
        // Do some processing on the raw value that ideally we'd like under unit test, and return it
        ...
    }
}

Then it's quite easy to write a test on the extension method, using a mocked or stubbed implementation of ISomething. Fairly obvious in hindsight - or maybe foresight you might be thinking! - but nonetheless seems something that might be easy to do (i.e. make an interface too wide, and thus make some code hard to test in the implementation of that interface).

Wednesday, 30 May 2018

Tackling Common Concerns with Azure Function Patterns

I've recently had (or perhaps am about to have if you have found this early) the pleasure of talking at the Intelligent Cloud Conference in Copenhagen, in May 2018, on the topic of Tackling Common Concerns with Azure Function Patterns

I've collated here a few links to resources discussed in my session.

First, find a copy of the slides here (in pptx, so you may find fonts a bit off without our company one), or, perhaps better, as a PDF here.

Then links to GitHub repos containing the full implementations of the patterns discussed in the talk, and from where the code samples presented are drawn.

Some tools mentioned in the talk:

Lastly, the Azure functions documentation.