Wednesday, January 21, 2015

Real-time Communication in AngularJS with ($Q)orlate

This post introduces the lightweight ($Q)orlate library.

A popular type of engagement that iVision is hired for is the architecture assessment. For assessment projects we analyze the existing system, weigh our own experience, industry standards and best practices against what we find, then provide feedback and a potential road map to iterate towards a desired solution. I’ve participated in everything from an over-arching analysis of the entire technology stack with associates from our data center, converged networking, and infrastructure practices to provide a business/IT roadmap for cloud readiness to a very targeted analysis of AngularJS implementations.

Recently I worked with a customer that built a very mature proof-of-concept using Angular in a system that had what I call “disconnected asynchronous processes.” These are processes that are truly “fire and sort of forget” because the message goes through message queues and workflows prior to a response being generated through a different channel. Although my first step is always to challenge this type of architecture (sometimes it’s implemented with the belief it is automatically faster or more scalable when there are other approaches that can achieve the same results with less complexity), in this case it truly seemed like the right architecture needed by that system.

I am constantly looking for opportunities to simplify and/or refactor code to reuse common components, algorithms, and strategies. This makes it easier to maintain and understand the code and provides developers with a toolbox to use so they can plug in the repeatable pattern and focus on what is unique about their part of the application. This led to me brainstorming what I’ve seen in these types of systems with Angular and I came up with three very distinct scenarios that I felt could be addressed with a lightweight Angular service.

1. Cached Requests

Angular has its own built-in mechanisms for controlling a cache. It is a common pattern for a service to asynchronously request information that is used by other components. Sometimes your app may get caught in a chicken-egg scenario when a component relies on a central service for information but must wait for that information to appear. For example, it may be necessary to retrieve security information from the server prior to building a menu or servicing requests.

There are a few ways to deal with this in Angular. One is to create the promise on the route, which will delay loading a view until the promise is satisfied. This isn’t an option if you aren’t using routes and sometimes your services may not be specific to a route. Another mechanism is simply to return a common promise. Multiple callbacks can be registered, so if the service keeps track of a single promise it can keep providing it, and even if it was resolved in the past the promise will be satisfied.

To see what I mean, take a look at this jsfiddle. Notice the second call happens long after the list was already populated, but the promise still fires and populates the second list (the list is populated after 1 second, and the second promise is requested after 2 seconds). The same promise is always returned.

The following figure shows this scenario (you can assume the server simply calls out to the database, the queue and handlers will make more sense for the other scenarios).

cachedrequests

To enhance this scenario, ($Q)orlate provides an option to track multiple requests with timeouts. If timeout isn’t a concern or is handled locally, the straightforward use of $q will suffice. What ($Q)orlate does is provide a mechanism to automatically timeout and expire the request, so it keeps track of multiple requests even if you satisfy them with a single call.

The call to ($Q)orlate is passed an id that uniquely identifies the correlation (it will generate one for you if you do not pass one in). In this example a timeout is used to emulate the delayed response. Notice the resolution has an extra parameter for the correlation.

$timeout(function () {
    _this.list = [1, 2, 3, 4, 5];
    _this.qorlate.resolve(_this.id, _this.list);                    
}, qorlate.defaultTimeout + 100);

The service always returns a correlation. This is because qorlate will immediately return the last promise that was resolved or rejected, so any values (success or error) will be satisfied in the promise.

return this.qorlate({ id: this.id }).promise;

The consumers simply call the service with a standard promise, and it will either be satisfied once the data is loaded or rejected when it times out (in the example, the initial call times out but the subsequent call works because the data has been loaded). ($Q)orlate guarantees the promise, so even if a request is rejected from a timeout and the original call returns, later requests will be resolved immediately with the result.

Although this is one use case, ($Q)orlate is more suited to two other scenarios.

2. Discrete Disconnected Messages

I am purposefully using the label “disconnected” instead of “asynchronous” because a direct AJAX call can be asynchronous while it is still “connected” in the sense there is a clear path from the call to the resolution of the promise. What is less clear is when the message is sent to be processed, then returns through another channel such as a websocket. The top section of the following figure demonstrates this:

CorrelatedRequests

I call this a correlated event because even though the return event is surfaced independently of the request (i.e. via a Web Sockets connection), it is correlated to the initial request. For example, you might have a form that submits to a queuing system, is validated by a handler on the backend, then the response is routed back. In these cases, you still want to keep track of the message (i.e. if it times out it is probably good to inform the user) but the API might not be so clear. Consider something like this that you may be used to:

var id = 1;
function sendRequest() {
    service.on('response', function (response) {
        if (response.id === id) {
            // do something
            // and don't forget to unregister
        }
    }
    service.send(id, request);
}

Basically you must keep track of your correlation and register to listen to all returned events, parse out the one you are interested in, then remember to unregister from the event (or just let it fire assuming the identifier won’t repeat), then finally fire off your message.

One of the nice things about the promise specification is that it helps make code cleaner and easier to read. What if you could do something like this instead:

service.request(id, request).
    then(function success(response) {
        // yes!
    }, function error(err) {
        // nope!
    });

This is a straight promise. Is that possible, even when the mechanisms are disconnected? The answer is, “Yes.” The service simply needs to establish the promise with a correlation:

var correlation = this.qorlate();
return
correlation.promise;

When the message comes in, the correlation is notified:

this.qorlate.resolve(correlation.id, response);

You can also reject the correlation, and the configurable timeout will also reject it automatically if the return message is not received within the timeout period. In the above example, the service generated the identifier. If you have GUIDs or other ways of matching messages, you can specify them when you generate the correlation and match it back. See a working example that demonstrates successful, failed, and timed out correlations here and view the source here.

If you look back at the second figure you’ll see there is another section near the bottom that indicates alerts raised by the server completely independent of the client. These are “broadcast” messages. You can certainly use mechanisms built into scope, but if you want a way that doesn’t rely on scope, still participates in digest loops (so no matter the mechanism for notification, you are always guaranteed to notify your subscribers in the context of a digest loop), and is testable, consider the next scenario.

3. Event Aggregator or Pub/Sub

The idea behind an event aggregator is straightforward. A central service “brokers” messages. You simply post your message to the service, any component interested subscribes, then the message is broadcast to the subscribers.

The subscription looks like this:

var cancel = q({subscribe:'addItem'}).always(function (item) {
    rs.list.push(item);
});

In this case, the subscriber is responding any time the message is sent. An error/rejection method may also be added so if there is a problem with the transport it will notify subscribers. I do realize always is used in a different context relative to promises, but it made sense here because that’s what is happening – the message is always notifying the subscriber.

The call to ($Q)orlate will return a function to cancel the subscription – that’s as easy as calling cancel(); and you’re unsubscribed. The API to raise an event is no different:

q.resolve('addItem', (new Date()).getTime());

In fact, as you can see from the test specifications, you can overload the same identifier with promises and subscriptions. The module is fairly lightweight. It:

  • Creates a promise when the function is called
  • Sets a timer to reject the promise if the timeout is hit (this is configurable at the provider level and the individual call)
  • Tracks the promises so when the correlation is resolved or rejected it can notify all consumers
  • Generates a fresh set of promises for recurring subscriptions

Because it is all based on the underlying $q service, you do not have to worry about a return call being generated outside of a digest loop as $q is integrated with the root scope.

You can browse the source, offer feedback, generate pull requests, read the full documentation, run the tests and read the samples online at https://github.com/jeremylikness/qorlate. Enjoy!

Wednesday, January 14, 2015

Web API Design Jumpstart

New to Web API design? Or experienced but want to know more?

Expert Christopher Harrison and I have the information you need. Join us on February 4 for "Web API Design Jump Start." In this informative and practical Web Wednesdays event, you'll learn why it is the ideal way to surface APIs that target browsers and mobile devices. Want a closer look at token-based security features, route attributes, error handling, and versioning? Whether you're brand new to the framework or you want to take your design to the next level, this course has the details you need.

You'll explore Web API technology, uses, and nuances. And you'll learn how the toolset makes it easy to build consumable RESTful services, accessible by a variety of clients from myriad platforms. Check it out!

Course Outline

  • Introduction
  • Basic Design
  • Configuration
  • Validation and Error Handling
  • Security
  • Advanced Design

Register now!

Web API Design Jump Start

Date: February 4, 2015

Time: 9am‒5pm PST

Where: Live, online virtual classroom

Cost: Free!

Registration: http://www.microsoftvirtualacademy.com/liveevents/web-api-design-jump-start

I look forward to "seeing" you there.

Jeremy Likness

Sunday, December 28, 2014

The Top 5 Mistakes AngularJS Developers Make Part 5: Failing to Test

Welcome to the last installment in a series of posts about the popular framework AngularJS. This series is designed to help you avoid common traps and pitfalls before they become a problem.

The top five mistakes I see people make are:

  1. Heavy reliance on $scope (not using controller as) 
  2. Abusing $watch
  3. Overusing $broadcast and $emit
  4. Hacking the DOM
  5. Failing to Test

Is failing to test really a problem with Angular, or a programming philosophy in general? I think it’s a little bit of both, and in this post I’ll explain why.

Failing to Test

If you’re not sure what testing has to do with Angular, take a look at their own developer guide entry for unit testing. I’ll just quote the following:

Angular is written with testability in mind, but it still requires that you do the right thing.

That’s a pretty clear directive (some of you see what I did there). First, I’d like to walk through how Angular embraces testing. Then, I’ll explore why it matters and how it’s not just a philosophical decision when you’re building large Angular apps. Then I’ll wrap with some final thoughts. If that works for you, keep reading.

Angular = Testing

I am pretty confident with that statement. Let’s explore the facts:

  • The framework was written to be testable
  • Dependency injection is a large part of the framework and helps make unit testing easier
  • Karma was written specifically to make running tests easier
  • Angular provides a module called ngMock right out of the box to make testing easier by injecting and mocking services such as $http
  • Protractor. Seriously.

So What?

Some of you will say, “So what?” Just because you can do something doesn’t mean you should, right? Fortunately, most of my decisions in programming are pragmatic and based on real world experience, not hypothetical pontification. I’ve not only worked on Angular projects that didn’t have tests, or other projects that did have tests, but have the experience of being on large projects that started without tests and then introduced them so I have a pretty good idea of the impact.

And it’s all positive.

There are some obvious benefits of tests you’ll hear thrown around like:

  • Find defects as early as possible (haven’t you heard, bugs are more expensive to fix in production?)
  • Refactoring assistance – don’t you feel better about changing code when you know you can run some tests to figure out the side effects right away?
  • Automated regression – as your application evolves, the tests make it easier to determine how your code base is keeping up with the changes

For me, though, the real benefits of testing aren’t so obvious until you’ve experienced them firsthand.

Requirements Refinement

Believe it or not, a good testing strategy can improve the quality of your requirements. This is essential when you are managing a large application. In fact, one of my favorite projects involved a collaborative feedback loop between the business, testers, and coders (you know, that “agile thing”) and helped really refine requirements to something usable.

We made it simple: when defining a backlog item, there would be testable acceptance criteria. We’d work with the team to ensure that criteria really could translate to a test.

So, you might start with something like this:

“Given a user when they enter their information then the system should calculate some fitness data.”

Not a very good requirement, is it? We could get more specific and state,

“Given a user when they enter their information then the system should compute their basal metabolic rate.”

That’s better, but I still can’t write a test. How about a few examples? Here’s one that is specific:

“Given a 40-year old 5 ft. 10 in. male who weighs 200 pounds when BMR is calculated then should compute 1929.”

Wow! Now that I can write a test for. In fact, I did write one you can execute here and it looks like this:

describe("Formula for BMR", function () {
    describe("Given a 40-year old 5 ft 10 in male who weighs 200 pounds",
        function () {
            it("should compute a BMR of 1929",
                function () {
            var actual = formulaBmr({
                isMale: true,
                height: 70,
                weight: 200,
                age: 40
            });
            expect(actual).toEqual(1929);
        });
    });
});

Of course I added a few other scenarios. Having  requirements with explicit acceptance criteria that translate directly to tests will completely transform how you deliver enterprise software.

API Design

The second impact of testing is that it directly impacts the design of your API. I’m not talking about afterthought, after-the-fact tests you write just so you can check a box that says, “I wrote tests.” I’m talking true test-driven development (TDD) that involves writing the test first, watching it fail, then writing the simplest piece of code you can to satisfy the test. You should try it sometime. It takes a lot of patience at first to change the way you approach it, but I found it creates clean, readable, maintainable code.

In the example above, for example, I originally had an interface that took several parameters for the calculation. I quickly realized it wasn’t evident looking at the call what was being passed or even what order. So I updated the API to take a parameter object instead with clearly defined labels (in hindsight I should have been even more explicit and made it heightInches and weightPounds).

That resulted in writing a service that looks like this:

function formulaBmr(profile) {
 
    // Women - 655 + (4.35 x weight in pounds) + (4.7 x height in inches) - (4.7 x age in years)
    function woman(weight, height, age) {
        return Math.floor(655 + (4.35 * weight) + (4.7 * height) - (4.7 * age));     }
    // Men - 66 + (6.23 x weight in pounds) + (12.7 x height in inches) - (6.8 x age in years )
    function man(weight, height, age) {
        return Math.floor(66 + (6.23 * weight) + (12.7 * height) - (6.8 * age));     }
    return profile.isMale ? man(profile.weight, profile.height, profile.age) :
        woman(profile.weight, profile.height, profile.age); }

Notice it’s a standalone, clean JavaScript object I could use in any application. It only becomes “Angularized” when I register it with dependency injection:

(function (app) {
    app.factory('formulaBmrService',
        function () {
        return formulaBmr;
    });
})(angular.module('healthApp'));

In fact, following a test-driven approach almost always results in clean, modular, portable code. In many cases I find that 80% of the code isn’t tied to a specific framework but exists as domain objects that perform certain tasks, and only get pulled into a framework to resolve dependencies and participate in data-binding.

Program Design

Testing does more than just improve API design. When I set out to build a 6502 emulator, I had no idea where to begin. So, I started writing tests to make the CPU do what I wanted. The CPU wouldn’t really do much without software to make it run, and most example programs for the 6502 8-bit chip are written in assembly language that must be compiled to machine code. To make this happen I wrote a set of compiler specs so I could load programs and test them in the emulator. Eventually I got the emulator up and running. To see it in action, simply load a sample program, compile, then run it.

Tests can help drive the design of your overall app, and that’s exactly how I wrote the emulator.

Documentation

The final not-so-obvious advantage to tests is that they document the system. If you’re not sure how “op codes” work for the 6502 chip, just take a look at the op code tests. It should be clear what the various labels stand for, what their function is, and what the expected result is for various operations. Even end users can read well-written test specifications and understand how a system is supposed to work.

It doesn’t stop there! Developers can read the source code of the specs to determine how the various APIs work. The example I linked to demonstrates the various flags that exist in the 6502 CPU and how they operate. A test can therefore drive design at multiple levels, provide documentation of requirements, and demonstrate exactly how to consume the components of the system.

Test-Driven Development

TDD is an approach that requires discipline, patience, and most importantly buy-in to implement if your team does not already embrace it. Even if you decide you do not want to follow a TDD approach, I strongly encourage you to make testing a part of your development lifecycle. Don’t just write tests to check a box or make a code coverage number, but instead integrate them as part of a meaningful approach to reap the benefits I’ve listed.

One technique I found is useful for adopting tests is to take a two-fold approach. The first is with requirements. Ensure requirements contain acceptance criteria that is specific enough to write tests for, and if you write no other tests, at least write the tests that will satisfy acceptance criteria.

Another way to integrate testing is through your defects resolution cycle. If a bug is entered into the system, the first step should be to write a test that fails because of the bug. You won’t believe how effective this simple approach can be if you aren’t already following it.

First, being able to duplicate the bug requires that you are able to gather enough information to reproduce it. This places the onus on your testing team to collect the information you need. It also requires intimate knowledge of the code so you know what components to write the test against that cause the failure. If you find it is difficult to write tests for the majority of bugs, you might need to take a step back and assess your architecture. The same design that makes it difficult to test could be contributing to the instability that is causing defects to surface in the first place.

Creating the test helps you get more familiar with the code that is related to the defect, and also creates a baseline for preventing future defects. Once you have a failing test, you can work to modify the code so the test passes. After it passes you can pass on the fix with confidence it was properly addressed, and slowly build a test suite that ensures the defects don’t reappear as side effects to later changes.

If you want an example of a full test-driven development cycle to build a reference application, check out my post entitled Let’s Build an Angular App! The deck will walk you through a series of github commits that represent an app that was written with a test-driven approach. You can see it evolve and watch how I took simple steps to layer components together that ultimately contributed to the final result.

That’s All He Wrote!

I saved this post for last because I believe the first 4 mistakes are actually easier to avoid when you take a test-driven approach. The Angular practices I’ve picked up over the past few years are based on hands-on experience architecting systems to scale with large teams and make it easy to integrate new features. When the team is focused on a test-driven approach, they tend to create solutions that:

As you can see, the top 5 mistakes are really all related! I hope this series helped improve your approach to providing business solutions with Angular. Let me know your thoughts and please feel free to add any of your own tips or common mistakes in the comments below! I look forward to hearing from you and appreciate all of you who have contributed your questions, thoughts, suggestions and tips to my previous posts in this series.

signature[3]