nathanblack.org

Refining Modern Web Development

Monday, January 30th 2017

Don't Mock Your Code - The Behavior is the Unit

I first started really utilizing unit testing and test-driven development while on the DataMarket team at Microsoft was slowing migrating an ASP.NET WebForms app to MVC. In a team that had no tests maintained by developers (shudder how did we develop software this long?) it was a big cultural change led by my great colleages Jon Sequiera and Dmitry Frenkel, who I am very greatful to have the experience of working with.

At the time we'd write tests for our controllers. We'd mock out all the dependencies it would need in a test and provide them. We were also using the repository pattern and testing the controller we'd mock that out. The repository pattern was really needed as we couldn't actually make a call to the database due to a lot of different reasons that basically stem from a monolithic ball of sphagetti code and abomination that was Entity Framework 1.0.

That was nice to get developers used to writing tests for their code, but there was a lot of problems with that. Views were not tested at all, due to Razor being untestable at the time (some 32bit vs 64bit issue on the build server or something, see previous comment on monolithic ball of sphagetti). We had bugs in our view not being rendered. Razor was also very logic heavy, and a few abominations were created. Writing new tests required about 3x the mocking code than code that actually did the assertions. Model binding had no tests - you'd know about it 3 days later (maybe) when a tester ran their test suite, or worse, your manager was trying something on the development environment. Forgot a dependency injection registration that you mocked out? App won't boot at all despite your tests passing. Moreover, when I want to refactor code to move more logic into the controller from the repository that had to much (that had no tests on it), I have to refactor a whole quite of tests. How can we rely on these tests when it still has these huge gaps and maintence problems?

Sunday, January 29th 2017

Migration to a static site: Experience with Metalsmith vs Gulp

I started this site using poet on top of node.js and express, mainly wanting to expand my node skills and use my free Azure credits I got as a Microsoft employee. Azure Websites, the Heroku-like PaaS had just launched (now called App Services). It was a nice experience.

Later on while working at Appuri, we decided to move our public website off of Hubspot as it was a nightmare for our talented designer, Jeff Reynolds, who knew HTML and CSS, to work in. It would need marketing functionality, but we took the approach that whatever we needed could be provided via a 3rd party JavaScript tag on the client. So we didn't need a server at all. We could simply serve it from an S3 bucket cheaply and it will always be up. Circle CI would build the site and sync the bucket.

So how to build it?

Saturday, December 10th 2016

Semantic Web Component Design: Radios, dropdowns, and tabs are the same thing

Early on in the rewrite of Appuri's portal that we started on after having a designer and development team from the MVP, and in our transition from Angular to Knockout, I was looking around at different widget / web component libraries and what I could leverage that would fit the designer's use cases and style, as well as a clean design.

One thing that always drove me nuts about our previous UI was a mismash of different components doing the same thing. We had ui-select2, ui-select from the Angular UI team, angular-multi-select for some multiselect cases, and a custom directive for radio-like toggle boxes.

Each control offered some little feature that another didn't (except ui-select2 which was just in code we didn't get around to refactoring yet). Why is this? All of them are trying to do the same thing on a higher semantic level: The user needs to pick an one (or more) options from this fixed set of options. How that looks is a styling concern.

Thursday, September 17th 2015

A fast and robust way to run Sauce Labs tests in Circle CI

We've been using Circle CI at Appuri for many months now and have been very happy with it. It's very simple and has a great breadth of features, namely the ability to SSH into a build to diagnose it when things get really tricky, as well as docker build services, slack integration, and lots of built-in services. After I hit a wall with PhantomJS, and needing to run our tests on multiple browsers anyways, it was time to integrate Sauce Labs to do our browser testing, which I've had great experience with on many open source projects. In those projects I used grunt-saucelabs as I am not a big fan of Karma as I like to control the HTML page the tests run on as well as the simplicity of mocha-phantomjs and just refreshing a flat file or using browser sync.

However we're using gulp and I didn't want to pull in grunt and have two build systems. Unfortunately there's no gulp-saucelabs, which I contemplated doing (I may sometime), but I found I could simply use mocha-cloud. However, it didn't bring up the tunnel.

Saturday, July 11th 2015

jspm and systemjs: High friction package managment

We have a decent-sized Angular codebase here at Appuri that I want to replace with real module system that will allow for a gradual move away from Angular. A friend recently pointed me to jspm and systemjs. The goal and direction is sound - load any module format (AMD, CommonJS, Harmony/ES6) and be registry-agnostic (npm or GitHub is out of the box with registry plugins, namely for bower). Unlike bower, since the loader and the package manager work together, it maintains the mapping of modules that normally you do manually in the map and paths sections of your RequireJS main. It also has a builder, so you can work in a mode that loads files individually, or bundle them up together for production, with source maps and minifcation. It also supports ES6 compiling to ES5 (or "transpiling" if you don't fully understand what compiling is), so it is very future-standards focused. It's a superset of what I want from RequireJS, so I was excited and dove in.

Three days in, I'm ready to throw in the towel as I can't get SystemJS builder to work. There are bigger issues too.

  1. Bower should be a first class registry

    So jspm is registry agnostic, and it's registry providers built in are npm and GitHub. Why a GitHub provder instead of bower? It's so similar to bower, yet lacking too. They are both git based (well except these muts be in GitHub, damn your priviate or alternate git repo). It also ignores dependencies in the bower.json - which basically defeats the purpose of the package manager as you are now left to dealing with the dependecies yourself, manually editing your config.js - the thing that it's supposed to manage for you.

    I started out simply using jspm install angular. Worked fine. Then trying to install angular plugins, they weren't in the registry. so I did them mangually via jspm install github:somedude/angular-plugin. However they would often just have a bower.json, so the dependency on angular wasn't declared. It was worse as I did angular-chart, which depends on c3, which then depends on d3.

    Ugh. So I install jspm-bower-endpoint and run jspm install bower:angular-chart. cool.... wait, it depends on bower:angular, but I already have github:angular installed from the jspm registry... Ugh. jspm uninstall ALL the jspm packages and just use the bower registry, otherwise unless the jspm registry is 100% complete, you'll just have a mess to manually maintain. The same is true if you mixed and matched npm with bower, unless the dependencies didn't cross. BTW, you still need to have a package.json to get jspm down anyways.... so you have 2 package maangers. Actually, 2 package managers and a package manager registry wrapper.

  2. Automatic module detection really does not work

Friday, May 15th 2015

Angular's module system fails at tracking dependencies properly

Angular, being a framework with a lot of solutions to a problem, forces users to use modules - a very good thing. However, rather than letting users choose their module and package system (RequireJS or Browserify or Webpack for loaders and AMD vs CommonJS vs Harmony for module formats), it brings in it's own module format, without a loader.

Defining a module looks like this

var module = angular.module('fruit-stand', ['raspberries', 'strawberries', 'apples'])
// define what the module is or has by calling methods on it

it also includes a dependency injection component, because, uh, that's what they use in Java so JavaScript needs one too, because modules can't be scoped or replaced in a dynamic, interpretted language right? Nevermind. Well here's how it works:

Sunday, June 1st 2014

Browserify string transforms: Use rfileify

I started to use browserify for the first time day. It is quite slick - Just npm install something then require it up, with a lot less wiring up needed than using AMD. However adding a forced build step is not fun and I think would really be a killer on a big project.

So I began to look for a transform so I could include templates as I needed to bundle everything into one script. How hard can it be to basically do "module.exports = " + JSON.stringify(fileContents) ? Well after a couple hours of searching and trying 3 modules, the 4th one worked.

Rfileify does other transforms too, but just not the SEO rank, so hopefully I can help that.

Wednesday, May 28th 2014

Why node.js is Awesome, Part 2: Package and Dependency Management

At the core of node is npm, the node package manager (which apparently npm doesn't stand for that). Having dependencies managed by a package system is huge. In Java and .NET, package managers have been added after the fact, if they are used at all. Often people end up checking in binaries that then makes the repository huge with information not pertinent to your product, which also results in binary merge conflicts. How do those get organized? A legacy codebase I work on has them scattered throughout 3 different locations, not to mention naming styles.

Npm standardizes this by having all the dependencies in a folder named node_modules which cannot be renamed and a package.json that must be comformant too, which then node understands when you require something. No need to set classpaths or hint paths correctly.

This also heavily pushes people to have all their dependies locally. Its still possible to have a global module installed and require it without declaring it in your package.json (Maybe there is a way to prevent this via static analysis?) You can end up pulling assemblies from the GAC in .NET too, but still so many packages aren't installable via NuGet (I'm looking at you Azure SDK, which of course the Azure SDK for Node.js installs via npm) which means every developer and the build server(s) better have the same versions installed or someone will be wasting time getting stuff to build.

Dependencies are also a tree structure, not a graph. If two modules depend on the same module, they each get their own copy, so they can have their own versions. This is great given prototypical inheritance. If you want to be strict about the type of object you are working with, instanceof will only be true if your version of the module created the object, not another version , due to the constructor functions not being a reference to the same object. If you want to be lenient, you can always duck type it.

Statically typed languages have a graph of dependencies, and if A and B depend on different versions of C, pain will ensue. I gave seen .NET developers waste hours tracking down mismatching versions of dependencies, even with the help of assembly binding redirects. I don't think Java has any answer for this.

Sunday, April 27th 2014

The 4th aspect in JavaScript MVC frameworks

Of the many JavaScript frameworks out there, they all have one core priciple - to structure your code in a Model-View-Whatever style:

  • Model: Your data as an object
  • Controller/ViewModel/Presenter/Whatever: the business logic of your app
  • View: The HTML templates for your app

    But after using a few (Knockout, Angular, Ember, and Backbone) I've noticed there's a forth component. It's the widgety type of logic - logic that isn't about your business, but extents the browser functionality - that cool datepicker, color picker, an awesome visualization with D3, maybe a 3D flip switch intstead of a checkbox. That stuff that is a DMZ to the DOM - Knockout's bindingHandlers, Angular's directives, and Handlebar helpers. They aren't about how you map your view model / controller to the DOM, but about extending the functionality of the DOM.

    Why call this out? Well frameworks don't exactly agree on this terminology. With Knockout and Angular, the view is HTML, as they use DOM-based templating. In Ember, Backbone, and other string-based templating languages, they often refer to the view as a piece of JavaScript, and templating is separate. Really the template should be called the view in their world, and the view is this 4th aspect, the widget aspect. Knockout and Angular have their bindingHandlers / directives that extend their templating functionality, which are much more analogous to Ember/Backbone/Batman/etc views.

    Why is this important? Keeping selector strings out of your controllers and view models is vital for the separation of concerns and tesability. Keeping this 4th aspect as the DMZ to the DOM helps to separate these concerns. Say you do have a complicated problem to solve in your app, and you come up with a cool way to visuallize it. You could put it all in one module... but now what happens when you release it and users don't like that widget and they want something more traditional? It will be harder to pull it out. Or the opposite where that widget is really awesome and you want to use it elsewhere... now lots of work to pull the business logic out. Let those two evolve independently and they grow and improve much more rapidly.

Friday, April 4th 2014

Why node.js is Awesome, Part 1: The Single Responsibility Principle

My favorite thing about node.js and the npm community is something that can't easily be put on a spec sheet. It's adherance to the single responsiblity principle.

Take a look at some of the most depnended on npm modules - they're often very small, like mkdirp, which is simply mkdir -p in node. glob matches files like a shell does. node-uuid just generates UUIDs.

Is there a bug in the module? The module is small enough and tested well that you can dive in and help out, and send the bug fix as a pull request on GitHub. Don't like the design of the module your using, or development died out? Swap it out with a different module, and wire it up to your loosely coupled system.

This is definately a change in culture, as though I didn't use Rails much, but that community likes it's big frameworks that do a lot of magic. Which is nice until you disagree with an aspect of that magic, or need to tweak it or swap it out with something else for this one scenario.

A great coworker of mine once told me "The key to complexity is composition", which I have experienced to be true.

Sunday, March 30th 2014

The Beginning

After owning nathanblack.org for several years and doing web development, it's long overdue I have a blog with a few thoughts to share about web development.

This blog is powered by poet on top of node.js and express.js, with stylus for the CSS preprocessor, but most importantly no jQuery and no bootstrap.

Update: See the entry about the current setup