Everything that went wrong when deploying my Node + Hapi app to Microsoft Azure

A couple of weeks ago I switched this website (https://nozzlegear.com) over from C# + ASP.NET to Node + Hapi + TypeScript + React. I'll say this right off the bat: there was no valuable or practical business reason to make this switch; it was just a fun little experiment to help me get a handle on Hapi and deploying a Node app to Microsoft's Azure.

I had three goals for this transition:

1. Switch the tech stack to 100% JS/TS with React-rendered views, which was positively delightful. Seriously, I love being able to write an entire app using only JS for both the frontend and backend.
2. Completely remove this website's reliance on a database for storing blog posts, and instead rely on simple markdown files. When I want to publish a new blog post, all I have to do is push the new post to the site's GitHub repository and Azure will automatically build and deploy it thanks to continuous integration. This is ideal for me as a developer, because I don't need to build or rely on a CMS system for creating/editing blog posts.
3. Learn more about deploying Node apps to Azure. Azure has been my preferred deployment target for a few years now, but being a primarily C#/ASP.NET consultant, I've never had a chance to deploy a Node app to it.

Goals #1 and #2 went extremely well! An app built with 100% TypeScript using React as its view engine is something I can't stop gushing about, and ripping out the database in favor of straight markdown files deployed with a simple git push is the cherry on top. (I'll be open-sourcing the base blog and writing a full walkthrough for those that want to try their hands at Node + Hapi + TypeScript + React very soon.)

Unfortunately, deploying this site to Azure was an absolute nightmare. Everything that could go wrong did go wrong. What follows is a monument to the complete frustration and rage that I felt over the nearly 3 hours it took to debug Azure's deployment process and get this website working.

The build process.

All Azure web apps run a background service called Kudu, which (among other things) automates application deployment for several different languages and frameworks including ASP.NET, Python, PHP and Node. If you're using Azure's continuous integration — which I highly recommend, deployments don't get much easier than git push origin master — it's Kudu's job to listen for new deployments, bring them in, build them and then transfer the built files over to the web app's site root.

For example, if you're running an ASP.NET project, Kudu will look for your packages.config file and make sure all of your Nuget packages are installed and up-to-date. After that Kudu will run MSBuild on your project file and finally copy all of the files over to the site root thereby updating the live site. The same goes for a Node project: Kudu will look for your package.json, run npm install, copy the files over to the root and start the server by running whatever start command you have listed in your project.json.

The web app I deployed (this website) is written in 100% TypeScript, so I don't check any JS files into the repository. For those not familiar with JS or TypeScript, Node can't actually run TypeScript files. They need to be compiled down to the base JavaScript before it can work with them.

The process for building all of the files to run this app goes like this:

1. Compile the TypeScript files with the tsc command.
2. Use gulp to both Webpack the few JS files used on the frontend and minify the CSS files.
3. Start the server with node bin/server.js.

To reduce that all into one single command, I just set the project.json's start command to tsc & gulp & node bin/server.js. That way I can run all three build steps with one simple npm start.

With my super simple build process ready to go, I flipped the switch and merged the pull request that converted this website from ASP.NET to Node. I reclined in my chair as Kudu received the changes and started to build it, pontificating on what a delightful experience this entire project had been.

And then everything broke.

Azure won't run multiple commands at the same time.

When Kudu looks in your package.json file for the start command, it takes whatever you have there and tries to run it in a Windows command prompt. The problem is that Azure apparently can't do concatenated commands, it can only do one command at a time. That means my tsc & gulp & node bin/server.js will not work on Azure and instead throws an error about invalid characters and commands.

(Note that running npm start in Windows itself will run those multiple commands just fine, so there might be something else going on here. Regardless, I couldn't find any documentation or reports for this so I have assume Azure just can't do concatenated commands.)

It was clear that I had to reduce my npm start script to just the single command that would start the server, but my app still needed to be built. I didn't want to start checking in my built files and node_modules folder, so after a little digging I discovered you can actually create your own Kudu deployment script and control your build process from there.

First, I edited the package.json's start command to just run the server with node bin/server.js, and then I added a build command that runs tsc & gulp so I could at least keep using a single npm run build on my dev machine.

Then I had to generate my custom Kudu deployment script. After installing the Azure CLI from NPM with npm install -g azure-cli I was able to quickly create a basic bash deployment with azure site deploymentscript --node --scriptType bash. (You can also create a Windows batch script with --scriptType batch. Even though I have extensive experience with C# and my main machine runs Windows, I'm more familiar with bash than batch.)

That command spits out a .deployment file and a deploy.sh file which looks like this. You'll need both of them in your project's root directory before Kudu will run your custom deployment script.

The default deploy.sh file for a Node server has three steps to it:

1. Copy all of the files in your repository to the site root.

2. Select a version of Node and NPM dictated by your package and website settings (more on this in a moment).

3. Install NPM packages in your site root.

I needed to edit that process so that Kudu would install my TypeScript definition files with typings, build my TypeScript files and run my Gulp tasks before copying files over to the site root. Here's what my modified Kudu deploy.sh file looked like after making those changes:

...

# Deployment
# ----------

echo Handling node.js deployment.

# 1. Select node version
selectNodeVersion

#2. Install NPM packages to repo root to run Typings, TSC and Gulp tasks
echo "Installing npm packages in KUDU repo root."
eval $NPM_CMD install
exitWithMessageOnError "npm install failed"
echo "Finished installing npm packages in KUDU repo root."

#3. Install typescript definitions
TYPINGS="node_modules/typings/dist/bin.js"

echo "Installing TS definitions."
"$TYPINGS" install
exitWithMessageOnError "Could not run 'typings'. Did 'npm install' run OK?"
echo "Finished installing TS definitions."

#4. Run TSC
TSC="node_modules/typescript/bin/tsc"

echo "Building Typescript files."
"$TSC"
exitWithMessageOnError "Could not run 'tsc'.  Did 'npm install' run OK?"
echo "Finished Typescript build."

#5. Run GULP
GULP="node_modules/gulp/bin/gulp.js"

echo "Running GULP default task."
"$GULP"
exitWithMessageOnError "Could not run 'gulp'. Did 'npm install' run OK?"
echo "Finished running gulp default task."

# 6. KuduSync => Copies files to deployment target
if [[ "$IN_PLACE_DEPLOYMENT" -ne "1" ]]; then
  "$KUDU_SYNC_CMD" -v 50 -f "$DEPLOYMENT_SOURCE" -t "$DEPLOYMENT_TARGET" -n "$NEXT_MANIFEST_PATH" -p "$PREVIOUS_MANIFEST_PATH" -i ".git;.hg;.deployment;deploy.sh;node_modules;"
  exitWithMessageOnError "Kudu Sync failed"
fi

#7. Install npm packages to deployment target
if [ -e "$DEPLOYMENT_TARGET/package.json" ]; then
  cd "$DEPLOYMENT_TARGET"
  eval $NPM_CMD install
  exitWithMessageOnError "npm site root install failed"
  cd - > /dev/null
fi

...

Note: I'm installing Node packages twice — once in the repo root, and once in the site root — because the KuduSync command which copies files over to the site root ignores the node_modules folder by default. Traditionally Windows has had problems with extremely long file paths in the node_modules folder, so I thought it was best to leave that ignore on there and just install the packages to the site root directly.

Kudu and NPM will only install production packages.

At this point my custom deployment script should now build the TypeScript files, run the Gulp task and then copy everything over to the site root where everything should just work. After adding the new deployment script files to my GitHub repository and pushing the changes to the master branch, Kudu took the new changes, found my custom deployment script, tried to run it and... threw an error.

Node's package.json lets you specify modules that your app depends on in both "production" mode and "development" mode (dependencies and devDependencies). Since Gulp, TypeScript, Webpack etc. aren't used in production (i.e. the website never uses those tools once it's up and running), I save them as dev dependencies; they're only used to build the production files.

It turns out that every Azure web app automatically sets an environment variable named NODE_ENV to production. It's a great way to programatically determine whether your app is running on a development machine or a live server (var isLive = process.env.NODE_ENV === 'production'). According to the NPM docs, NPM checks to see if that specific variable is set to production, and if so, it will only install production packages.

Now this is something that most Node developers probably know, but having never previously deployed a Node app to production, I was unfamiliar with this behavior. While Azure does let you change the NODE_ENV variable by editing the web app's Application Settings, that's not a wise thing to do. Plenty of code relies on that variable to determine specific behavior.

To fix this problem and get my TypeScript/Gulp build running, I had to make sure dev packages get installed to the Kudu repo folder before running those tasks. Luckily NPM lets you override which packages get installed with npm install --only=[prod|dev]. My webpack and minification Gulp tasks take files from production packages, so I had to make sure both dev and production packages were installed.

...

# Deployment
# ----------

echo Handling node.js deployment.

# 1. Select node version
selectNodeVersion

#2. Install NPM packages to repo root to run Typings, TSC and Gulp tasks
echo "Installing dev and production packages in KUDU repo root."
# Install production dependencies in KUDU repo root. Webpack + uglify gulp tasks reference prod packages,
# so dev + prod packages must be available to gulp.
eval $NPM_CMD install --no-progress --only=prod
# Force npm to install dev dependencies. Necessary because setting NODE_ENV
# to production will make npm skip dev dependencies on install.
eval $NPM_CMD install --no-progress --only=dev
exitWithMessageOnError "npm install failed"
echo "Finished installing dev and production packages in KUDU repo root."

#3. Install typescript definitions
TYPINGS="node_modules/typings/dist/bin.js"

echo "Installing TS definitions."
"$TYPINGS" install
exitWithMessageOnError "Could not run 'typings'. Did 'npm install' run OK?"
echo "Finished installing TS definitions."

#4. Run TSC
TSC="node_modules/typescript/bin/tsc"

echo "Building Typescript files."
"$TSC"
exitWithMessageOnError "Could not run 'tsc'.  Did 'npm install' run OK?"
echo "Finished Typescript build."

#5. Run GULP
GULP="node_modules/gulp/bin/gulp.js"

echo "Running GULP default task."
"$GULP"
exitWithMessageOnError "Could not run 'gulp'. Did 'npm install' run OK?"
echo "Finished running gulp default task."

# 6. KuduSync => Copies files to deployment target
if [[ "$IN_PLACE_DEPLOYMENT" -ne "1" ]]; then
  "$KUDU_SYNC_CMD" -v 50 -f "$DEPLOYMENT_SOURCE" -t "$DEPLOYMENT_TARGET" -n "$NEXT_MANIFEST_PATH" -p "$PREVIOUS_MANIFEST_PATH" -i ".git;.hg;.deployment;deploy.sh;node_modules;"
  exitWithMessageOnError "Kudu Sync failed"
fi

#7. Install npm packages to deployment target
if [ -e "$DEPLOYMENT_TARGET/package.json" ]; then
  cd "$DEPLOYMENT_TARGET"
  eval $NPM_CMD install --no-progress --only=prod
  exitWithMessageOnError "npm prod install failed"
  cd - > /dev/null
fi

...

Your selected Node version may not actually be your selected Node version.

Another commit and another deploy. So far my super simple Node + Hapi app isn't so simple anymore. In between each fix was about 30 minutes spent waiting for npm install to run (it's slower than molasses on Azure), trying to figure out what went wrong, and then attempting to fix it.

My saving grace was Kudu's service dashboard that runs on every Azure web app. You can access it by browsing to https://my-azure-domain.scm.azurewebsites.net. Among other things, this dashboard contains a file directory with your site's files and a PowerShell terminal to manipulate the site.

After my last commit, the site actually started to build! The TypeScript files compiled just fine and I thought I would finally be able to put this stuff behind me and move on to the next project. And then my Gulp webpack task exited with an error. Deployment was cancelled and I was more irritated than before.

The worst part was the errors didn't even make sense. Webpack was complaining that certain modules and file paths didn't exist, but I was able to verify that they did in fact exist using the PowerShell terminal. Thinking this was just a weird, random error, I used the PowerShell terminal to run npm install again and then called gulp webpack to run the webpack task.

Again, I got the same "module not found" error that killed the deployment. Several attempts later and nearing the end of my rope, I quickly checked the current Node version installed on my web app with node -v.

0.10.18

What. That's not right at all. It was using one of the oldest possible versions of Node and NPM possible, a version that my Gulp tasks weren't supported on.

It was running 0.10.18 despite specifically setting my desired Node version to 5.4.1 in my package.json's engines property — a setting that the Kudu docs say will be honored. Not only that, I've also set the WEBSITE_NODE_DEFAULT_VERSION environment variable to 5.4.1 in my web app's Application Settings — another one that the docs say will be honored.

I thought about running npm install npm -g to force the latest version of NPM, and doing similar for Node itself, but I really didn't want to monkey around with the web app's virtual machine potentially breaking things.

Extremely frustrated at this point and about to file a bug with Kudu, I noticed the Kudu service dashboard listed a "Runtime versions" API endpoint. You can reach it at https://my-azure-domain.scm.azurewebsitse.net/api/diagnostics/runtime. This endpoint actually lists all of the Node runtime versions that you can use with your Azure web app.

Version 5.4.1 was not on that list, which is why my web app was defaulting to 0.10.18. However, 5.4.0 is on that list, and after switching both the package.json's engines field and the WEBSITE_NODE_DEFAULT_VERSION environment variable over to 5.4.0, my Gulp tasks were finally able to complete.

Google Font imports don't work when minifying CSS with clean-css on Azure.

With Azure now using the version of Node that I thought it was using, it was now time for the next problem. Thankfully this is a short (but weird) one. When my Gulp task minifies CSS, it (apparently) tries to process all font imports by downloading the files directly.

When you try to do this from a virtual machine on Azure, though, you get this error:

Error: Broken @import declaration of "https://fonts.googleapis.com/css?family=Open+Sans:300,400,600" - timeout

While there doesn't seem to be a clear cause for what happens here, it's suspected that Google is throttling requests from bot-like traffic. Being in a virtual machine on Azure, this request almost certainly looked like a bot.

The fix for this one was very simple, just set processImport to false when minifying via Gulp:

gulp.task("minify", function ()
{
    var cssMinOptions = {
        processImport: false
    };

    var task = gulp.src([...])
        .pipe(cleanCSS(cssMinOptions))
        .pipe(gulp.dest(...));

    return task;
});

I didn't know which port my Hapi app should be listening on.

Another commit and push to Azure, and now my deployments were finally building. TypeScript was compiling my tasks, Gulp was running webpack and minify, and Kudu was copying the built files over to the site root. This website had now been officially converted from C# + ASP.NET to Node + Hapi!

I loaded up my website expecting to see essentially the same thing that was there before (just powered by JavaScript) and I got... nothing. Just an error screen telling me the resource I requested could not be found.

When you're writing the code for a Node server app — whether it's running Hapi, Express or anything else — you need to tell it which port it should be listening on to serve requests. For example, if I set my Hapi app to listen on port 3000, then I can load it up in the browser at http://localhost:3000.

The problem is Azure doesn't know it should forward requests to your localhost:3000 server. So even if your server app is happily chugging along, diligently listening for requests on whatever port you gave it, it will never receive a request.

I wasn't able to determine whether you can tell Azure to forward requests to a certain port, but thankfully there's a better way to get requests to your Node app. Instead of you telling Azure which port configuration to use, Azure tells you.

Azure sets two other environment variables named PORT and HOST, which you then plug into your app to get requests forwarded from Azure to your server app:

...
const isLive = process.env.NODE_ENV === "production";

const config: Hapi.IServerConnectionOptions = {
    port: process.env.PORT || 3000,
    host: process.env.HOST || "localhost",
    router: {
        isCaseSensitive: false,
        stripTrailingSlash: true
    }
};

const connection = server.connection(config);

server.register([plugin1, plugin2, ...]).then(() =>
{
    ...
})
.then(() =>
{
    ...

    return server.start((error) =>
    {
        if (error)
        {
            throw error;
        }

        console.log(`${isLive ? "Live" : "Development"} server running at:`, server.info.uri);
    });
});

And that's the chronicle of my nightmare deployment to Azure. In the end, it all worked out (or else you wouldn't be reading this), and I'm extremely satisfied with the app itself despite the frustration caused by the deployment. No doubt much of my trouble could have been avoided if I had previous experience in deploying a Node app to production, but we all have to start somewhere.

I'll be opening up the code for this blog soon, and will likely write about the actual development process too.

Finally, I'm planning a new tutorial series on building at least 3 brand new Shopify applications that heavily utilize Typescript and React with a Node + Hapi backend. If you've bought my book on building Shopify apps with C#, this series will be a great introduction to building Shopify applications with the world's most popular programming language.

Enter your email below and I'll make sure you hear about those tutorials the second they're done.