Monday, August 31, 2009

Silverlight Captcha Example

Silverlight, with its powerful text and graphics manipulation capabilities and strong interaction with the scripting DOM, seems to be the perfect engine for a Captcha challenge.

See an Example Online

Download the Source (18.3KB)

Captcha is a challenge-response test used to determine to a degree of confidence that the end user is really human and not a bot. You see Captcha on things like forums, sign up forms, comment posts, and other places that may be succeptible to attacks by scripted bots. Usually, a Captcha involves playing a sound or displaying a distorted image that humans should be able to recognize but would be extremely hard for pattern-matching and/or optical character recognition (OCR) technology to decipher. Because the test is issued by a computer to test for a human, it is often referred to as a reverse-Turing test (a Turing test is designed by humans to test computers).

The key to Captcha is that it makes it difficult, if not impossible, for scripted software to determine the answer to the challenge. Asirra is an example of a Captcha challenge that displays random images of cats and dogs. The user is asked to identify only the cats by clicking on them. While easy for humans, this test is extremely difficult for computer algorithms.

As I was coding the other day, it occurred to me that Silverlight would be perfect for issuing Captcha challenges. It is very easy and straightforward to manipulate text and graphics to obscure the image, and furthermore, the output is NOT a simple bitmap that a robot could parse. Instead, it is an interactive plugin so for a script to recognize the image, it would have to have its own Silverlight engine and be able to scan and recognize what Silverlight renders.

I set out to produce a working example. I purposefully kept to the basics so those of you reading this who are interested have the opportunity to extend and add features.

The first step was to create a simple Captcha challenge class to use.

namespace SilverCaptcha.MVVM
{
    [ScriptableType]
    public class CaptchaViewModel
    {
        private const string CAPTCHA_KEY = "SilverCaptcha";

        private static readonly char[] _charArray = "ABCEFGHJKLMNPRSTUVWXYZ2346789".ToCharArray();

        public string CaptchaText { get; set; }

        public CaptchaViewModel()
        {
            char[] captcha = new char[8];

            Random random = new Random();

            for (int x = 0; x < captcha.Length; x++)
            {
                captcha[x] = _charArray[random.Next(_charArray.Length)];
            }

            CaptchaText = new string(captcha);

            HtmlPage.RegisterScriptableObject(CAPTCHA_KEY, this);
        }

        [ScriptableMember]
        public bool IsHuman(string challengeResponse)
        {
            return challengeResponse.Trim().ToUpper().Equals(CaptchaText);
        }
    }
}

The class simply generates a random 8-digit sequence of characters. We supply a list of allowed values to avoid some of the common characters like the number one and letter "I" that could be easily mistaken for one or the other. The property CaptchaText exposes this value. The IsHuman method is decorated with the ScriptableMember tag. This makes it available to the Html DOM so that you can call it directly from JavaScript. To call it, you must register a "key" or handle to the object. This is done in the constructor through the RegisterScriptableObject call. We are giving it a handle in the JavaScript DOM of "SilverCaptcha." We'll see later how this is used.

Points of Extension

  • Add a method to "refresh" the challenge, i.e. generate a new one (hint: to do this, you'll also need to implement INotifyPropertyChanged)
  • Add parameters to control the challenge (length, use of alpha or numeric, etc)

Next, we'll need to show the challenge. Because I chose to use the Model-View-ViewModel pattern (MVVM) we won't need any code behind for the XAML. Instead, everything will be bound in the XAML itself. The XAML I came up with looks like this:

<UserControl x:Class="SilverCaptcha.MainPage"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" 
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:MVVM="clr-namespace:SilverCaptcha.MVVM">
    <UserControl.Resources>
        <MVVM:CaptchaViewModel x:Key="CaptchaVM"/>
    </UserControl.Resources>    
    <Grid Width="100" Height="25" Margin="2" DataContext="{StaticResource CaptchaVM}">
        <Grid.Background>
            <LinearGradientBrush x:Name="CaptchaBackground" EndPoint="0.5,1" StartPoint="0.5,0">
                <GradientStop Color="LightBlue" Offset="1" />
                <GradientStop Color="LightSalmon" />
            </LinearGradientBrush>
        </Grid.Background>
        <TextBlock FontSize="14" Width="Auto" Height="Auto" 
                   HorizontalAlignment="Center" VerticalAlignment="Center" 
                   Text="{Binding CaptchaText}"
                   RenderTransformOrigin="0.5,0.5"> 
            <TextBlock.RenderTransform>
               <RotateTransform Angle="-5"/>              
            </TextBlock.RenderTransform>
            <TextBlock.Foreground>
                <LinearGradientBrush EndPoint="0.5,0" 
                    StartPoint="0.5,1">
                    <GradientStop Color="Black" Offset="1" />
                <GradientStop Color="Gray" />
                </LinearGradientBrush>
                                    
            </TextBlock.Foreground>
        </TextBlock>
    </Grid>
</UserControl>

This is fairly straightforward. By referring to CaptchaViewModel in the resources, an instance is created that can then be referenced using the key. I bind the class to the data context of the main grid using {StaticResource CaptchaVM}. The gradient is used to obscure the image somewhat. Because the class itself is bound to the grid, we can now simply bind the CaptchaText property to the text block. We also give that a slight gradient to make it more confusing to image scanning software, then rotate it against the background. That's all there is to it!

Points of Extension

  • Obviously, you could randomize or parameterize the angle of rotation and other attributes of the text
  • A more severe grid may help obscure the challenge but may also make it more difficult for human readers
  • Add a refresh button or icon to refresh the challenge for the user

Now, let's use it in a page. The page itself is fairly straightforward: we have a form with a table, and inside the table is a reference to the Silverlight XAP file, a text box for the user to enter their response to the challenge, and a button to click. This section looks like this:

<form id="_form1" runat="server" style="height:100%">
    <table><tr><td align="center">
    <div id="silverlightControlHost">
        <object id="silverlightControl" data="data:application/x-silverlight-2," type="application/x-silverlight-2" width="110" height="30">
        ... etc, etc ...
     </object></div></td></tr><tr>
     <td align="center">Enter The Text You See Above:<br /><input type="text" maxlength="20" id="txtChallenge" /></td></tr><tr>
     <td align="center"><input type="button" onclick="validateCaptcha();return false;" value=" OK "/></td></tr></table>
    </form>

(Click here to see a live example)

The key here is that we've assigned the silverlight object an identifier of silverlightControl. If you use the Javascript or control method to load the silverlight, either way you just need a way to point to the object in the DOM for the Silverlight control.

The JavaScript function is then very straightforward. We simply call the method we exposed in the Silverlight class that will compare the response to the challenge. That code looks like this:

function validateCaptcha() {
    var silverlightCtrl = document.getElementById("silverlightControl");
    var challenge = document.getElementById("txtChallenge").value;
    var valid = silverlightCtrl.Content.SilverCaptcha.IsHuman(challenge);
    alert(valid ? "You are human!" : "You don't appear to be human, try again!");
}

This is what the final product looks like:

Captcha Challenge

As you can see, calling the Silverlight method is as simple as grabbing the control that hosts Silverlight, going into Content, then referencing the tag we gave it when we registered it in the Silverlight code. Then we simply call the method with the contents of the text box and it returns either true or false based on whether or not the response matches the challenge.

This is obviously a fast and basic example but it demonstrates just how flexible and powerful Silverlight can be to generate "mini-apps" or controls that you can embed in your existing HTML pages.

See an Example Online

Download the Source (18.3KB)

Jeremy Likness

Wednesday, August 26, 2009

Windows Workflow Sequential Workflow Unit Testing

Windows Workflow Foundation (WWF) is a powerful programming model and designer for workflow processes. Using workflow is easier than some may believe.

I wanted to share a quick post relative to some practices I've found useful for sequential workflows, specifically around Inversion of Control (IoC) and unit testing.

In our shop we have several complex algorithms that are great candidates for sequential workflow. By using sequential workflow, we can:

  • Easily visualize the overall layout and flow of the process
  • Test the architecture of the flow (i.e. do we get to this node or that node) without having to worry with or deal with the underlying implementation of nodes within the flow
  • Refactor complex code simply by dragging and dropping nodes in the designer
  • Setup unit tests to ensure the integrity of the workflow persists as other systems are refactored

If you aren't familiar with workflow, I'd suggest searching for a few tutorials or walkthroughs. This is one of those technologies that is probably easier to learn by building some test projects than it is simply reading about it. The scope of this article will be how to build your sequential workflows for unit tests.

Interfaces and Delegates In

The workflow model lets you declare public properties on the main workflow that are then available to all of the nodes within that workflow. When you instantiate the workflow, you can send in a collection of name-value pairs map to public properties. For example, if I have an interface declared on my workflow:

...
public IMyInterface MyInterfaceImplemented { get; set; }
...

Then I can pass to workflow an instance of that interface like this:

...
Dictionary<string, object> myParams = new Dictionary<string, object>
   {
      {"MyInterfaceImplemented", new MyInterfaceImplementation()}
   };
...
_runtime.CreateWorkflow(typeof (MyWorkflow), myParams);

I'll get to what _runtime is in a bit. My point is, however, that any type of business logic you follow within your workflow should be hidden behind an interface and/or delegate. If you are passing in concrete instances, you are missing out on the power of being able to unit test your workflow structure.

You may have noticed that the code blocks in a sequential workflow, unless encapsulated as a custom entity, are actually events triggered by the workflow engine. For example, I might insert a Code workflow activity called AnalyzeUser. In the code behind, this appears as:

private void AnalyzeUser_ExecuteCode(object sender, EventArgs e)
{
}

In some cases you might be calling a method or service on an existing interface. However, other cases may be simple snippets of code. In order to keep your workflow completely pluggable, consider using a delegate. For example, if the analyze user takes a user parameter and then sets a flag in the workflow based on the result, instead of this:

private void AnalyzeUser_ExecuteCode(object sender, EventArgs e)
{
   _isOver30 = User.Age > 30;            
}

Try this:

// declare a delegate to analyze the user
public delegate bool AnalyzeUserDelegate(User user); 

// declare a method to set the delegate
public AnalyzeUserDelegate AnalyzeUserMethod { get; set; }

private void AnalyzeUser_ExecuteCode(object sender, EventArgs e)
{
   _isOver30 = AnalyzeUserMethod(User); 
}

Note we've encapsulated the action that is required (set a flag by analyzing the user) but we've removed the dependency of a concrete implementation. Why would I want to do this? Perhaps I am testing the overall workflow structure and want to know when it reaches the call to the delegate. I'm not concerned about whether it sets the users age, only if the method is reached. I can easily test for that situation like this:

...
bool delegatedCalled = false;

AnalyzeUserDelegate testMethod = (user) =>
   {
      delegatedCalled = true;
      return true;
   };

myParams.Add("AnalyzeUserMethod", testMethod);
...
instance.Start(); // start the worfklow
Assert.IsTrue(delegateCalled, "The method was invoked."); 
...

This allows you to separate testing the logic/flow of the workflow from the implementation of the various nodes within the workflow.

Properties Out

We talked about how to get implementation into the workflow, now let's talk about what to get out. Even if your workflow does not "return" a value to the outside world, you should consider exposing some properties to indicate key milestones within the workflow. One of our workflows involves assigning an asset to a location. The workflow itself only needs to perform the actual assignment. There is no "output" other than the changes to the system itself. However, for testing purposes, we expose certain values such as the location assigned and a flag that indicates whether an assignment was successful. This allows us to test the outcome, like this:

...
public bool WasAssigned { get; set; }
...
instance.Start(); // start the workflow
Assert.IsTrue((bool)results["WasAssigned"]); 

That example is a simple test of whether or not a successful assignment took place.

Wiring the Tests

One thing I've noticed is that few people realize workflows are very easy to test. There are many ways to trigger a workflow (events, schedules, etc). The standard examples I've seen online involve launching the engine in potentially a different thread altogether. Fortunately, the engine is extensible and there is a service shipped with the runtime that allows you to run it in an almost-synchronous way.

The first thing to do is set up references to the run time and the service that allows for the immediate workflow execution:

...
private WorkflowRuntime _runtime; 
private ManualWorkflowSchedulerService _scheduler; 
...

In your test setup, you'll instantiate the runtime and add the manual service:

[TestInitialize]
public void TestInitialize()
{
   _scheduler = new ManualWorkflowSchedulerService(); 
   _runtime = new WorkflowRuntime(); 
   _runtime.AddService(_scheduler); 
}

Be sure to tear down the workflow after the test is run:

[TestCleanup]
public void TestCleanup()
{
    if (_runtime != null && _runtime.IsStarted)
    {
        _runtime.StopRuntime();
    }

    if (_runtime != null)
    {
        _runtime.Dispose();
    }
}

Now you can wire up your test. If you want to be able test the output results as I mentioned above, you need to bind to the WorkflowCompleted event:

...
Dictionary<string, object> results = null;
_runtime.WorkflowCompleted += delegate(object sender, WorkflowCompletedEventArgs wce) { results = wce.OutputParameters; };
...

This will take all public properties and dump them by name and value into the results variable for inspection. The next thing is simply to create an instance and run it, like this:

WorkflowInstance instance = _runtime.CreateWorkflow(typeof (MyWorkflow), myParams);
instance.Start();
ManualWorkflowSchedulerService schedulerService = _runtime.GetService<ManualWorkflowSchedulerService>();
schedulerService.RunWorkflow(instance.InstanceId);
Assert.IsNotNull(results);

The results test ensures the workflow successfully ran.

Now I can test my workflow structure using a mocking framework like Moq. Moq makes it easy to inject a test object and verify whether or not it was called. If I have an interface called IMyUserServices and I am expecting the workflow to call a method called ValidateUser, I would simply load it up like this:

Mock<IMyUserServices> myUserServices = new Mock<IMyUserServices>(); 
Dictionary<string, object> myParams = new Dictionary<string, object> 
   {
      { "MyUserServicesImplemented", myUserServices.Object }
   };
// set up the call to always return true
myUserServices.Setup(svc => svc.ValidateUser(It.IsAny<User>())).Returns(true); 
...
instance.Start();
...
// make sure we called it
myUserServices.Verify(svc => svc.ValidateUser(It.IsAny<User>()), Times.Once());
...

Obviously going through all of the features and functionality of mock frameworks is outside of the scope of this post, but I wanted to share how easy these tools make it to create powerful, granular tests. I am able to wire up the entire complex workflow and test that the flow works as expected before every writing a line of actual implementation. Then, it becomes simple to implement the code and write unit tests for each atomic piece of functionality. This then makes it easy to test and implement complex solutions by breaking them into simple parts.

Finally, I'll leave you with a little snippet of unit testing wisdom I recently learned. A stub is something you inject to make a test run. A mock is something you check the state of to validate your test. Often people refer to stubs as mocks, but if you aren't inspecting the stub for success, it's a stub. If you are checking it for something (state, output, exception) then it is a mock.

Jeremy Likness

Tuesday, August 25, 2009

Uncommon Common Table Expressions (CTEs)

Common table expressions (CTEs) were introduced with SQL 2005. They are a powerful new construct for building advanced queries.

The Microsoft CTE Article describes 4 main uses for CTEs:

  • Recursive queries
  • Inline views
  • Grouping by derived columns
  • Reuse of the same table

It still amazes me how many people on the new SQL platform actually aren't familiar with or don't use CTEs. I'm going to share two quick-and-dirty examples to demonstrate just how powerful they can be.

Hierarchical Data

I've blogged a bit about hierarchical data in the past (read Hierarchical Data Templates in Silverlight for an example). SQL makes it far easier to retrieve data in a hierarchical fashion using CTEs.

Let's assume you have a content management site with articles, and articles can be grouped into hierarchical categories (i.e. one category may contain others). There are several ways to approach this, but one way is to have the main category table referenced by another that encapsulates the relationships, like this:

Category Example

We'll start with an Articles section that has Whitepapers as a subcategory, which also has "Technology Related" as a subcategory, then add a FAQ which has "Using the Site" and "Logging Out" as subcategories. Our main table looks like this:

CategoryID Category
1Articles
2FAQ
3Whitepapers
4Using the Site
5Logging Out
7Technology Related

To show the relationships, we'll enter records into CategoryLevel like this:

Parent Child
13
24
25
37

One real-world challenge would be showing a breadcrumb when the user is viewing a technology article. Given the child node (7, Technology Related) how do we recursively traverse the hierarchy to show all of the parent nodes?

Using a common table expression, we would alias the "top" of our query (the selection of the current category) and then join that to recursively traverse the tree, like this:

with RecursiveCat(CategoryID,Category) 
 as (
  select
    CategoryID,
    Category
  from dbo.Categories
  where CategoryID = 7
  union all 
  select
    L.ParentCategoryID as CategoryID, 
    Categories.Category 
  from dbo.CategoryLevels L 
  join RecursiveCat c 
    on c.CategoryID = L.ChildCategoryID
  inner join dbo.Categories 
    on L.parentCategoryID = Categories.CategoryID
 )
 select * from RecursiveCat

The key here is to note the "union all" with the join. The top query "kicks it off" and selects the target node. The union describes a recursive template ... on the inside, where we reference the CategoryLevels table that describes the parent-child relationships, we join to the Common Table Expression (CTE) on the outside. This means for each record, it joins back to itself and "unrolls" until the full hierarchy is exposed.

Running the above query yields:

CategoryID Category
7Technology Related
3Whitepapers
1Articles

As it walks up the tree.

Ranked Data

The other most common use is for ranked data. Let's take the above example even further, and declare an articles table:

The articles simple belong to a category and have a date when they were published. I'm leaving out details like title, body, etc for now.

A very common way to feature articles on a content website is to show the most recent article published to a given category. Assuming you have dozens of articles, what is the easiest way to do that?

Of course, if you are pragmatic, you'll probably put a MostRecentArticleID column on the Categories table or even introduce a new table to hold that. However, if you want to have fun (and let me keep my excuse for giving an example) you can use a rank function (we're not yet to the Common Table Expression, so bear with me). For now, try to ignore my typo as ArticlePublishData should be ArticlePublishDate. The first query will give you an idea of how the RANK works:

SELECT CategoryID, ArticleID, ArticlePublishData,
     ROW_NUMBER() OVER (PARTITION BY CATEGORYID ORDER BY ArticlePublishData DESC) As "MyRank"
   FROM Articles 

When we run this, we get all of the articles, but if you notice, the "MyRank" column resets for each category. Within a category, it starts at the most date date with "1" and then increments. This is because we are partitioning, or telling the CTE to reset the ranking, by the categories, and then ordering within those categories by the publish date. In this example, both 1 and 2 are in the same category, then 3 is in a new category.

CategoryIDArticleID ArticlePublishData Rank
122008-12-13 00:00:00:00 1
112006-12-14 00:00:00:00 2
232009-01-02 00:00:00:00 1

Now, it's just a simple matter to add a filter using a Common Table Expression. We'll alias the ranking function to extract the row, and then filter on the row:

;with CTE(CategoryID, ArticleID, ArticlePublishData, Row) AS (
SELECT CategoryID, ArticleID, ArticlePublishData,
     ROW_NUMBER() OVER (PARTITION BY CATEGORYID ORDER BY ArticlePublishData DESC) As "MyRank"
   FROM Articles) 
SELECT * from CTE
where Row = 1

This will give us only the most recent article for each category.

Hopefully this has been a useful, quick, and easy introduction to Common Table Expressions. Just search online for the phrase and you'll find plenty of examples and tutorials to help you take your SQL skills to the next level.

Jeremy Likness

Saturday, August 22, 2009

More ah-hah

Listening to the talk by Rik Robinson and Steve Porter of Wintellect (this is at the Atlanta Silverlight Firestarter) ...

  • Ah-hah ... forget my earlier post on multiple values. May be obvious, but if you leave the property off the Binding tag, i.e. pass Binding instead of Binding Name, you get the whole object (slapping forehead)
  • Remember that binding is hierarchical. Instead of having to explicitly bind controls, Silverlight/WPF both let you bind at a higher level and reference at a lower. I knew this, but may be an ah-hah for others
  • Cross-domain policies require explicit references to subdomains, it is not assumed they are valid just because the root domain is the same
  • POX = plain old XML (as opposed to SOAP, REST, etc)
  • Ah-hah ... Rik likes a vertical hidden task bar
  • Important ah-hah ... WCF services return on the UI thread, allowing direct access. Other methods do not, and require that you use the Dispatcher to communicate across the threads
  • Use duplex WCF services for realtime applications ... very powerful (using a polling mechanism under the covers)!
  • AH-HAH! use the new browser stack instead of the client stack to expose the actual fault exceptions to the Silverlight clients (set in the service binding as well as specified ono the client) ... no customer service interceptor

Jeremy Likness

Some Basic Silverlight "Ah-hah" Moments

Just finished listening to Shawn Wildermuth discuss Silverlight "soup to nuts." It was good to get back to the basics. Despite having done several applications including LOB apps for our business, there were a few "ah-ha" moments during the presentation.

  • Ah-hah ... so resources are keyed, not named, to prevent the objects from being created multiple times. One reference can be used (may be honest, but was an ah-hah moment for me)
  • Ah-hah ... partial methods (if you haven't looked into these, very interesting) make a lot of sense for extending generated code. Note to self: look further into "why extend vs. inherit?"
  • Ah-hah ... so instead of building a complex validation framework, why not use NotifyOnValidationError and ValidatesOnExceptions, then simply modify the template if needed?
  • Ah-hah ... why roll so many of my own converters when there are now mature open source projects like SL Extensions to draw from?
  • Ah...hah ... cool Visual Studio add-in: Rock Scroll, replaces the scroll bar with a code preview

There are many more very exciting sessions coming up, I'm sure I'll post more as I can!

Jeremy Likness

Wednesday, August 19, 2009

Exception Expected? Assert.True ...

Awhile back, Ayende Rahien posted about Assert.True as a tool of last resort. It was a great post (if you're into testing, which if you're a developer, I'm assuming you are, you should read it) and I chimed in about a pattern that bothered me:

...
bool handled = false;

try 
{
   something;
}
catch(Exception ex)
{
   handled = true;
}

Assert.IsTrue(handled);
...

Tests are meant to ensure appropriate behavior. Our methods validate their contracts early by testing the parameters passed in. If I am expecting something not to be null, and you give me a null, it's my obligation as your friend to pass back an ArgumentNullException instead of letting it flow through until you get a NullReference.

Someone replied about a helper method and I had that "slap forehead" moment because I hadn't thought of this:

private delegate void ExceptionDelegate();

private static bool _ExceptionExpected(Type exceptionType, ExceptionDelegate method)
{
    bool retVal = false;

    try
    {
        method(); 
    }
    catch(Exception ex)
    {
        retVal = ex.GetType() == exceptionType;                
    }

    return retVal; 
}

...
Assert.IsTrue(_ExceptionExpected(typeof(ArgumentNullException), () => targetClass.TargetMethod(null)));

A lot cleaner and reusable ... and I'm confident someone reading this will come back with something even more elegant, but for now, I like it!

Jeremy Likness

Thursday, August 13, 2009

IoC Doesn't Have to Break your Build!

If you're not familiar with dependency injection or inversion of control, check out this older post by Jeremy Miller.

There are a million reasons why these patterns make sense and help you build better code. Many of the new guidance and patterns depend on, well, dependency injection (take the Composite Application Guidance AKA PRISM for example). One major object I've seen and heard is "it's great, but I can't use it ... it's going to break my build!"

The main reason for this is the fact that inversion of control depends to a large extent on constructor injection. Take for example your customer data access class that used to look like this:

public class CustomerDataAccess
{
   public void SaveCustomer(CustomerEntity customer) 
   {
      ConsoleLogger logger = new ConsoleLogger();
      SQLDataAccessLayer layer = new SQLDataAccessLayer(); 
      layer.Save(customer); 
      logger.Log("Saved the customer."); 
   }
}

After kicking it around for awhile you realized your logger, which used tracing, stopped working in the web project and one of your customers wanted the backend to be MySQL instead of SQL. Your class evolved to something like this:

public class CustomerDataAccess
{
   public void SaveCustomer(CustomerEntity customer) 
   {
      ILogger logger = LoggerFactory.GetLogger();
      IDataAccessLayer layer = DataAccessFactory.GetDataAccess();
      layer.Save(customer); 
      logger.Log("Saved the customer."); 
   }
}

It was easy because in your factory, you simply checked a few app.config settings and returned the appropriate instance.

Now DI and IoC come along and you want to plug in the Unit Framework. Only that means you are going to something like this:

public class CustomerDataAccess
{
   private ILogger _logger;
   private IDataAccessLayer _layer; 

   public CustomerDataAccess(ILogger logger, IDataAccessLayer _layer)
   {
      _layer = layer;
      _logger = logger;
   }

   public void SaveCustomer(CustomerEntity customer) 
   {
      _layer.Save(customer); 
      _logger.Log("Saved the customer."); 
   }
}

Suddenly, the 5000 places you make a data access layer break. You know you are going to go through and start getting an ICustomerDataAccess by called the inversion of control container, but do you really have to hold open the release and do it all at once?

The answer is, of course, no. While this may seem painfully obvious to some people, sometimes developers can't see the forest to the trees. They get too close to the code and thing it's all or nothing. Instead, you can simply do this:

...
public CustomerDataAccess() : this(LoggerFactory.GetLogger(), DataAccessFactory.GetDataAccess()) 
...

Now you've got your backwards-compatible behavior. You can then go into your factories and mark your methods as obsolete, and you'll receive compiler warnings where the old method is being used. Use this as your "to do" list and start wiring them up using the container instead!

Jeremy Likness

Wednesday, August 12, 2009

What's in Your Collection? Part 1 of 3: Interfaces

The collection is a powerful construct that allows a developer to logically group related elements and navigate through them. In .NET, a collection is specifically anything that implements the ICollection interface.

The entire series can be accessed here:

(If you enjoy this article, please vote for it at The Code Project)

The purpose of this series is to expose the collections provided with .NET in the System.Collections namespace and learn when and why you would use different types of collections. We'll understand the provided interfaces first, work through the concrete implementations available in the base .NET distribution then examine custom collections.

What's in a Bag?

No doubt you've dealt with collections in the past. There are many business requirements that relate to taking a group of similar elements and being able to parse, sort, and modify the set. The first such grouping most people learn about is an array, or an indexed list of items. Most languages have some sort of support for an array:

...
string[] myArray = new[] { "One", "Two", "Three" }; 
...

You can think of collections as being items in a bag, but this is not precise enough for our definition. In a bag, you can blindly reach in and grab something. A collection in .NET on the other hand allows you to enumerate through the items, so they must be indexed in a deterministic way. In this respect, it's more like a clothes rack: the clothes are hanging in a precise order and you can turn the rack and go through one piece of clothing to another. A single item is there only once, and it is a finite set of objects: you can count exactly how many exist and even tell me that one item is next to another.

Now we can take a look at the .NET supplied interfaces and start to come up with more concise definitions for different types of collections.

The System.Collections Interfaces

System.Collections Interfaces

It all Starts with Enumerator

As you can see, it really all "starts" with the interface IEnumerator. Something that implements this interface can move through the set of items. You can Reset() to the beginning, take a look at the Current() object, and get to the next one with MoveNext().

The popular for loop that exists in similar form in multiple languages really encapsulates the entire interface. A typical example looks like this:

For Loop as IEnumerator

As you can see, the concept is similar ... get me to the beginning, get me to the next, and always have a reference to the current.

Are you Enumerable?

If the answer to this question is "yes" then it means you implement IEnumerator. Classes that implement IEnumerable are willing to give away their IEnumerator via the GetEnumerator() method.

Notice that the dictionary has it's own IDictionaryEnumerator. This extends IEnumerator by adding the entry, key, and value. This brings us to another set of definitions that are important to understand:

Arrays and Lists are indexed collections. You can use an offset or integer value to indicate the "position in line." Dictionaries are Maps. Huh? A "map" basically maps one value to another. A dictionary is not so much concerned with what position someone has in line, but what are they holding in their hand? Alex maps to a hat, and Jane maps to a purse.

If you are looking to simply put some items together and move through them, then you'll be looking at lists, arrays, and similar constructs. If you are looking to map one type of object to another, then you are dealing with a dictionary (the definition goes with the word). The dictionary enumerator exposes the mapping: an "entry" (the word and its definition), the "key" (the word), and the "value" (the definition).

More on dictionaries later.

Finally, my ICollection!

Looking at the ICollection interface, you can see that collections are not the same as lists or dictionaries. A collection is a basic building block. What's important to note is that a collection, unless extended, is not indexed. I cannot say up front, "Give me the 3rd element." Instead, I must iterate through the collection using the IEnumerator provided by GetEnumerator() to find out what is available. The collection interface says I must:

  • Be able to copy my collection to an Array (then I'll have my index!)
  • Be able to provide a count of objects, so it is a deterministic set with a known, finite set of items
  • Optionally allow synchronization.

The synchronization aspect is important. Imagine maintaining a single queue of items in a multi-threaded application. As a naive example, consider holding a list of IP addresses in a web process that spawns a new thread every time someone connects. Since the list is stored in only one place (presumably as a static class accessible via the singleton pattern), there must be a way to add an item to the list without colliding with the other threads that are competing for the same resource. If my implementation can be synchronized, I return true for IsSynchronized() and provide SyncRoot() for synchronization. This is an object that acts as a mutex for access. In order synchronize access, I follow this pattern:

public void AddItemToCollection(ICollection collection, object item)
{
   lock (collection.SyncRoot())
   {
       // add item here - oops, can't do that with just ICollection, I'll
       // need something more ...
   }
}

Making a IList, Checking it Twice...

Now we get to one of the more commonly used collections: the collection that implements IList. A list is a collection, so by way of collection it is also enumerable, and therefore provides an enumerator. In addition, it can be synchronized and has a count. Let's explore what else a list can do:

  • You can add an item to the list without caring or needing to know where it gets added
  • You can clear a list and start over from scratch
  • You can see if a list already contains an item (and it is implied that add won't actually add it if the item already exists)
  • You can figure out the cardinal index of an item in the list, if it exists (so I really can ask for that "3rd item")
  • You can insert an object into the list at a specified index (instead of adding it to the tail end)
  • You can remove individual items from the list, by looking at the item or considering the position (index) of the item
  • Lists can be read-only and/or fixed size (i.e. you cannot add beyond a certain limit set when the list is created)

IList provides a powerful interface for collections that have plenty of flexibility around adding and removing items.

Have that IDictionary Handy?

The dictionary is very similar to the list, but instead of giving us an enumerator, it provides a dictionary enumerator. Because a dictionary is a map, instead of just dealing with items, we actually deal with keys and values (remember the analogy of words and definitions). The key for a list is the index, i.e. to grab an item from the middle of the list, you map it based on its position. A dictionary, on the other hand, can have a key of any type and they way you reference an item is by the key, rather than the index.

This makes for an interesting observation: a list can be thought of as a dictionary where the key is the cardinal index of the value in the list.

A Question of Equality?

The remaining interfaces relate to functions that help keep the lists organized by knowing whether or not items are duplicates and how items compare to each other.

IComparer provides a scalar comparison between two objects. Regardless of what the object is, implementing IComparer means you can have the concept of something being less than, equal to, or greater than another object. This comes in handy for sorting objects and the Compare signature of the method allows the developer to define just how the objects literally compare to each other.

IEqualityComparer, on the other hand, determines simply if two objects are equal: true or false. How you determine equality is up to you to implement in the Equals method.

Both IEqualityComparer and IHashCodeProvider define a method for returning a 32-bit integer, GetHashCode. Obviously, these interfaces are for collections that are stored as hash tables. A hash is an integer generated from a hash function that tries to generate a unique number for an object. It maps potentially complex data to a smaller fingerprint or signature that should strive to be as unique as possible (however, hash functions do not guarantee uniqueness and when two pieces of data generate the same hash, the result is referred to as a hash collision).

While .NET provides a built-in hash function for strings, the common algorithm for this will add character values to each other and ignore overflows (values that exceed the upper limit for integers). To illustrate this, build this simple console application:

using System;

namespace HashTest
{
    class Program
    {       
        static void Main(string[] args)
        {
            foreach(string str in args)
            {
                Console.WriteLine(string.Format("{0}\t{1}", (uint)str.GetHashCode(), str));
            }
        }
    }
}

You can then run it from the command line and pass unique strings, like this:

Hash Function Example

As you can see, each word generated a unique integer, regardless of the size of the word, and the same words ("of") generated the same integer. In this case, we did not have a collision. Hash functions help ease the look up of complicated classes and structures. You should take care in generating your hash function and try not to overcomplicate it.

For example, a class with contact information that contains a first and last name can probably return a hash that is either the hash code of the concatenated string, or the sum of the hash codes for the first name and last name. However, if the phone number is stored, then the phone number itself may become the hash! If you are holding a lot of information in a class that is persisted and has a unique key, then the hash may be as simple as the value of the key itself, rather than something that looks at other properties on the class.

Random Thoughts

To tie this section together, I put together a small console application that demonstrates an implementation of these interfaces. You'll find this is a rather unique approach to the collection, but it honors the interfaces 100%. What the program does is create a special enumerator that generates random integers. This is wrapped in a collection. The collection can have a fixed "size." Any instance of this type of collection in the same AppDomain will provide the same sequence of random numbers. This is because a random seed is generated, then stored, and used as the seed for the sequence.

The key here is to note the behavior of IEnumerator and realize that just because we can deal with it like a collection doesn't mean we have to have a fixed list of integers in memory - in this case, we exploit the random number algorithm and the fact that the same seed returns the same sequence of numbers. Also note how simply by implementing IEnumerable, the collection can suddenly be used in a foreach. Here is the code:

using System;
using System.Collections;

namespace Enumerable
{
    internal class Program
    {
        private sealed class RandomEnumerator : IEnumerator
        {
            private int _index;
            private readonly int _size;
            private static readonly int _seed;
            private int _currentValue;
            private Random _random;

            static RandomEnumerator()
            {
                Random seedGenerator = new Random();
                _seed = seedGenerator.Next();                
            }

            public RandomEnumerator(int size)
            {
                _size = size;
                _random = new Random(_seed);
            }

            public bool MoveNext()
            {
                _currentValue = _random.Next();
                _index++;

                bool retVal = _index <= _size;

                if (!retVal)
                {
                    Reset();
                }

                return retVal;
            }

            public void Reset()
            {
                _random = new Random(_seed);
                _index = 0;                
            }

            public object Current
            {
                get { return _currentValue; }
            }
        }

        public class RandomCollection : ICollection
        {
            private readonly IEnumerator _random;

            private readonly int _size;

            public RandomCollection() : this(5)
            {
            }

            public RandomCollection(int size)
            {
                _size = size;
                _random = new RandomEnumerator(size);
            }

            public IEnumerator GetEnumerator()
            {
                return _random;
            }

            public void CopyTo(Array array, int index)
            {
                GetEnumerator().Reset();
                while (GetEnumerator().MoveNext())
                {
                    array.SetValue(GetEnumerator().Current, index++);
                }
            }

            public int Count
            {
                get { return _size; }
            }

            public object SyncRoot
            {
                get { return null; }
            }

            public bool IsSynchronized
            {
                get { return false; }
            }
        }

        private static void Main()
        {
            RandomCollection col = new RandomCollection(7);

            foreach (int item in col)
            {
                Console.WriteLine(item);
            }

            Console.WriteLine();

            int[] copy = new int[col.Count];
            col.CopyTo(copy, 0);

            for (int x = 0; x < copy.Length; x++)
            {
                Console.WriteLine(copy[x]);
            }
        }
    }
}

Once you compile this program, you can disassemble it using ildasm.exe to see what is generated "behind the scenes." The first block of code after instantiating the collection is the foreach loop. Look at how it appears in IL:

IL_0011:  br.s       IL_0029
    IL_0013:  ldloc.s    CS$5$0000
    IL_0015:  callvirt   instance object [mscorlib]System.Collections.IEnumerator::get_Current()
    IL_001a:  unbox.any  [mscorlib]System.Int32
    IL_001f:  stloc.1
    IL_0020:  nop
    IL_0021:  ldloc.1
    IL_0022:  call       void [mscorlib]System.Console::WriteLine(int32)
    IL_0027:  nop
    IL_0028:  nop
    IL_0029:  ldloc.s    CS$5$0000
    IL_002b:  callvirt   instance bool [mscorlib]System.Collections.IEnumerator::MoveNext()
    IL_0030:  stloc.s    CS$4$0001
    IL_0032:  ldloc.s    CS$4$0001
    IL_0034:  brtrue.s   IL_0013
    IL_0036:  leave.s    IL_0055

You don't have to be an expert in IL to see the use of the IEnumerable interface. The foreach literally invokes the interface by referencing Current and then looping through a call to MoveNext() until it returns false.

One thing you'll notice is the unbox.any operation. This is because we are using the non-generic version of the interfaces. This forces the runtime to box and unbox any value types in the collection. Fortunately, the namespace System.Collections.Generic gives us IEnumerator<T> and ICollection<T> that allows us to strongly type our collection. This would allow the actual types to be referenced and remove the need to box and unbox when iterating the collection.

We've taken a look at the basic interfaces provided by System.Collections. In the next article, I will cover the various concrete implementations of collections that are available so we can discover when, where, and why we'd want to use different types of collections in our own projects.

(If you enjoyed this article, please vote for it at The Code Project)

Jeremy Likness

Tuesday, August 11, 2009

The Saga Continues ...

Was working on an interesting problem today related to file transfers. An original service just tossed the file our way. The new service can break it into pieces and send the pieces.

Of course, I could have easily just created a temporary file with a special extension and kept appending bytes until the last chunk came, then copy the file, but this felt a little "hacked." Is there a cleaner way to separate the need to piece something together from the separate concerns of receiving something on the wire and ultimately persisting it? What about exceptions? What if the last pieces comes first or we suddenly stop receiving pieces at all?

After digging around I realized we were dealing with the saga pattern. There is a great document online that gives an example of the saga pattern. To the best of my knowledge (and I admit I could have it wrong) a saga is a long-lived transaction that may involve multiple services and can ultimately reach some consensus on when the transaction is complete.

In this case, we might be having that conversation with the same service (the one uploading the file) but the fact is, the conversation is around one topic (the file) and has several states (hello ... the story ... goodbye). We "agree" we're done when the conversation is concluded (i.e. the file is uploaded) or the other person walks away (we don't receive another piece after a given amount of time ... something we also need to "agree" upon to end the transaction as "timed out."

What makes then even more interesting is the composite pattern applied to the target. It seems to me I should be able to define an IPiece<T> where T is something I'm dealing with composing over time. I can then create an IPieceAssembler<T,U> where T: IPiece<U> where U: class ... basically the assembly has an Append method and an Assemble method that returns T. Finally, I can have an IAssembleSaga that represents the long-lived transaction with a BeginSaga(IPiece<T>), ContinueSaga, and EndSaga that returns a T or AbortSaga that cleans up. The saga is responsible for persistence until end saga is called and then cleans itself up, and some controller on top of sage will end the saga.

Now we have something interesting that can be re-used elsewhere as well ... hmmm!

This was all very exciting until my mentor brought me back down to earth. I was ... (shhh, don't tell) over-engineering! He made a great point: why am I taking so much time to build interfaces and abstractions when I'm not even sure about the concrete implementation? Turns out a generic is useless if it's not used anywhere else, otherwise just point to the type. Agile says we incrementally do what we know and don't try too hard to plan ahead ... if we come to that second case we can always extract an interface, create a generic type and refactor then, right? And sure enough, when I focused on the concrete implementation I realized there was no saga. I ended up writing bytes to disk.

Jeremy Likness

Monday, August 10, 2009

A Better Pipeline

As a follow-up to my Pipeline and Yield in C# post from a few days ago, I just realized I can make a slightly better pipeline.

First, taking the base class for a node and changing this:

public void Register(IPhrase nextPhrase)
{
    _nextPhrase = nextPhrase;
 
}

To this ...

public void Register(IPhrase nextPhrase)
{
    if (_nextPhrase == null)
    {
        _nextPhrase = nextPhrase;
    }
    else
    {
        _nextPhrase.Register(nextPhrase);
    }
}

Then allows me to take loading up the pipeline from this:

public static string Execute(bool spanish, bool goodbye)
{            
    IPhrase root = null;
    IPhrase previous = null; 

    foreach(IPhrase phrase in _GetPhrase(spanish, goodbye))
    {
        if (root == null)
        {
            root = phrase;
        }
        else
        {
            previous.Register(phrase);
        }
        previous = phrase;
    }

    return root == null ? string.Empty : root.Execute(string.Empty);
}

To this...

public static string Execute(bool spanish, bool goodbye)
{            
    IPhrase root = null;
    
    // load up the pipeline
    foreach(IPhrase phrase in _GetPhrase(spanish, goodbye))
    {
        if (root == null)
        {
            root = phrase;
        }
        else
        {
            root.Register(phrase);
        }                
    }

    return root == null ? string.Empty : root.Execute(string.Empty);
}

Notice that I don't have to keep track of "previous" anymore. As long as I know the root (which I need to know anyway, because that's what starts the pipeline going), any registration simply follows down the links until there is a free space and attaches there.

Also a good note from my architecture mentor ... this example builds the pipeline every time. A more efficient implementation would realize there are 4 valid states and build the 4 states ahead of time and reuse them, rather than build out the entire pipeline every call.

Jeremy Likness

Cool Design Tools

Just wanted to share a few killer design tools I found online:

Jeremy Likness

Friday, August 7, 2009

The "mini" View Model in Silverlight

I received a lot of value reading this post: Mini-ViewModel ... covers MVVM, data binding, and more in a short, elegant post! (Like the business card).

Jeremy Likness

Thursday, August 6, 2009

Pipeline and Yield in C#

The pipeline pattern (sometimes also referred to as pipes and filters) has many useful applications. C# makes implementation even easier with the yield keyword.

Download the Source (5.42 KB)

Pipeline is similar to chain of responsibility. The main difference is that in the chain, each "link" passes something to the next until one knows what to do with it, then the process stops. In pipeline, something is passed to every chain in the link and potentially modified with each pass. This is where "pipes and filters" come into play because one link may filter while other links in the pipe may add.

I was coding this pattern for a project where we had a set of complex rules to generate a path for a file. It was really a pipeline because we start with a root path, then keep adding subfolders and structure based on certain rules, and finally filter the path to have clean characters and then save the file. As I was designing this little module, I was presented with an interesting challenge.

You can think of the pipeline in really two ways. One way is that you have a pipeline manager that is aware of all of the links in the pipeline and manages their interaction. In this scenario, you would simply have a List<T> and then the pipeline manager would iterate each node, taking the result and passing it to the next.

The second is that your pipeline manager simply "feeds" the pipe, but the execution is really managed by each node. In other words, each node will do something, then pass the result to the next node and so on. This more of the traditional view and is in line with the concept as it originated years ago in Unix machines with literal input and output pipes.

The problem I had was when wiring up the nodes on the pipe, many nodes were conditional. The challenge is that I have to keep track of the root node, because that's the call that starts the whole pipeline processing, as well as the "current" node because that's what I add the next node to.

My code was becoming a little convuluted trying to keep track of the current node, next node, etc. That's when I decided to turn to my friend yield. With yield, you can conditionally insert elements into an IEnumerable collection. This was perfect for me: I could simply check my condition, then yield return the new node else fall through to the next. My pipeline controller would then use a generic algorithm and simply iterate the collection and register each subsequent node with the previous.

The generic algorithm looks like this:

INode root = null;
INode previous = null; 

foreach(INode node in _GetNodes())
{
    if (root == null)
    {
        root = node;
    }
    else
    {
        previous.Register(node);
    }
    previous = node;
}

Basically, I keep track of the root to call it later and the previous node so I can register the next one. The "magic" happens in _GetNodes because it has a return type of IEnumerable<INode>.

In order to demonstrate this, I made a very simply example. My pipeline is a phrase pipeline. Each node takes in a string, adds something to it, then passes it to the next node. I have four phrases: "Hello," "Hola," "Goodbye," and "Adios." To demonstrate the conditional chaining in the pipeline, my controller will accept two booleans, one to determine whether or not I include Spanish in the phrase, and another to determine whether or not I say, "Goodbye."

The interface IPhrase declares a simple Execute method that takes in a string and returns a string. It also declares Register to register the next node in the pipeline.

public interface IPhrase
{
    string Execute(string input);
    void Register(IPhrase nextPhrase);
}

In keeping with SOLID and DRY principles, developers shouldn't be concerned with how each node is registered or even with the fact that they need to call the subsequent node. In order to facilitate this, I created a base class. The base class takes care of the Register method. It also grabs the Execute method and wires in the call to the next node. Because I don't allow you to override this (because then you'd have to call base.Execute to get the base behavior) I provide an abstract method called _Execute that you can use to do your work. This means that the derived methods only have to focus on _Execute ... there is no need to worry about whether or not another node was registered or even how to call that node. The base class looks like this:

public abstract class BasePhrase : IPhrase
{
    private IPhrase _nextPhrase;

    protected abstract string _Execute(string input);

    public string Execute(string input)
    {
        string retVal = _Execute(input);

        if (_nextPhrase != null)
        {
            retVal = _nextPhrase.Execute(retVal);
        }

        return retVal; 
    }

    public void Register(IPhrase nextPhrase)
    {
        _nextPhrase = nextPhrase;
    }
}

Notice how we call the abstract method to modify the string, then pass it on to the next node if it exists. Now a "link" in the pipeline chain can look like this:

public class PhraseHello : BasePhrase
{
    protected override string _Execute(string input)
    {
        return string.Format("{0}Hello ", input);
    }
}

As you can see, the PhraseHello node simply appends "Hello " to whatever string is passed to it. The PhrasePipeline ties this all together. It has an Execute method that takes in the booleans (do we want Spanish, and do we want Goodbyes?), wires up the pipeline and then returns the call from the root. The following diagram illustrates the layout (click on the image to see a more detailed/expanded view):

Pipeline Example

The actual controller method looks like this:

public static class PhrasePipeline
{       
    public static string Execute(bool spanish, bool goodbye)
    {            
        IPhrase root = null;
        IPhrase previous = null; 

        foreach(IPhrase phrase in _GetPhrase(spanish, goodbye))
        {
            if (root == null)
            {
                root = phrase;
            }
            else
            {
                previous.Register(phrase);
            }
            previous = phrase;
        }

        return root == null ? string.Empty : root.Execute(string.Empty);
    }
    
    private static IEnumerable<IPhrase> _GetPhrase(bool spanish, bool goodbye)
    {
        yield return new PhraseHello();

        if (spanish)
        {
            yield return new PhraseHola();
        }

        if (goodbye)
        {
            yield return new PhraseGoodbye();

            if (spanish)
            {
                yield return new PhraseAdios();
            }
        }
    }
}

As you can see, the static class simply takes in the conditions. It then iterates the call to _GetPhrase and registers each subsequent phrase modifier with the previous. Finally, it simply calls the root node if it exists (we could easily pass in a "root" string here instead of string.empty).

The _GetPhrase method demonstrates the power of the yield command. In a nutshell, we are given an enumerable collection that we can loop through with a foreach as you see in the main method. However, the way yield works is that the element is inserted into the collection, then control is passed back to the caller. When the caller asks for the next element, code resumes after the previous yield statement.

In our example, every call guarantees me at least one element: PhraseHello. The subsequent calls are conditioned. If "Spanish" is false but "goodbye" is true, the code first evaluates the expression, skips it, goes into the "goodbye" section, and returns the PhraseGoodbye node. Now our "pointer" is ready to check whether or not we include Spanish.

Implementing pipelines this way provides incredible flexibility. You can have nodes make decisions about inserting child nodes, or even pass the chain to various controllers and factories that make decisions and add links as well. This, combined with a practical application of the yield keyword makes the code clean, readable, and easily extendable.

Homework ... want to take it further? You can easily abstract the pipeline itself by having INode<T> (hint: this example would become IPhrase : INode<string>) and make a Controller<T> to manage it. Also, don't limit yourself by thinking T has to be something like a string or a class ... it can easily be it's own IEnumerable<U> as well!

Jeremy Likness