Hook yourself to CLOUD: September 2016

Friday, 23 September 2016

Tutorial: Building an Azure Template

An Azure Template is a JSON file that captures the infrastructure requirements of your application. An Azure Template defines a number of related resources that are required by your application. Each resource represents a manageable entity, such as an Azure Website, a virtual machine, a SQL database, an identity, a load balancer, or an autos-scaling rule, etc. When an Azure Template is deployed on Azure, Azure Resource Manager ensures that all resources are brought to the specified states so that your application runs under the exact desired environments.

In this tutorial, we’ll walk you through of steps of authoring and deploying a simple Azure Template. The template defines a web application with a frontend running on Azure Websites and a database running on Azure SQL Database. The goal of this tutorial is to walk you through the structure of an Azure Template. In your actual projects, you should use graphical UIs to edit your templates to avoid manual editing errors.

Preparing the environment

At the time when this blog is written, the easiest way to get started with Azure Template editing is to use Visual Studio with Azure Resource Manager Tools (Preview) for Visual Studio. In addition, you’ll also need an active Azure subscription. You can get a free trial subscription at www.azure.com.

Getting started

Azure provides a rich set of Azure Templates in its Azure Gallery. In this tutorial we’ll use an existing template from the gallery to get the editing experience started. But we’ll clear the template so that we can start from scratch.

In Visual Studio, select the FILE –> New –> Project… menu.
On the New Project dialog, select the Cloud –> Cloud App from the Azure Gallery template, enter a Name for your project, and then click the OK button.
On the Select Azure Template dialog, select the Website + SQL template, and click the OK button.
On the New ASP.NET Project dialog, select MVC template, and click the OK button.
Once the wizard completes, you’ll have an ASP.NET website and an Azure Template that is ready to be deployed to Azure. Under the solution’s DeploymentItems\Templates folder, you’ll find an Azure Template namedWebSiteSQLDatabase1.json. This template already contains all necessary resources for you to deploy your website to Azure. However, for the sake of learning, we’ll clear the template to a bare minimum.
Modify the template so the code looks like the following code list:
```
{
    "$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "location": "West US",
    "resources": [
    ]
}
```
In this minimum template file, the $schema property indicates the schema version of the template file. ThecontentVersion property indicates the version of the template file itself. The main element of the file is theresources array, which contains all the resources to be defined in the group.

NOTE: There are several other top-level properties such as parameters, variables, tags and outputs. We’ll cover some of these properties later in this tutorial.
Save the template. And then, in Solution Explorer, right-click on the template and select the Deploy Template…menu. On the Deploy Template dialog, select the Azure subscription, storage account, and deployment location you want to use, then click the Deploy button to deploy the template to Azure.

NOTE: Leave the Template parameters file field at its default value. We’ll come back to this later.

The deployment should succeed after a short delay. You can observe the deployment traces in the Outputwindow. Deployment of an Azure Template is an idempotent operation so that you can deploy the same template as many times as you like. You can think an Azure Template as a definition of the desired states of your application resources. And Azure will make sure the resources are brought to specified states when you deploy the template.

Defining a Resource

In previous section you’ve deployed an empty Azure Template to Azure. In this part you’ll add a SQL Database resource to the template. You define a new resource by adding a new child element to the resources array.

A resource definition is a JSON object with three mandatory properties: name, type and apiVersion. The nameproperty identifies the resource. And the type property and the apiVersion property combined specifies which version of what Resource Provider to use. Azure Resource Manager has an extensible architecture, in which different Resource Providers can be plugged into the system to provide capabilities of managing a variety type of resources. For example, for SQL Database servers, the corresponding Resource Provider type is Microsoft.Sql/servers, and the current apiVersion is “2.0”.

Modify your Azure Template to include the definition of a SQL Database server resource. In the following code list, a new SQL Database server named mysqlserver has been added to the resources array.
```
 1:{
 2:     "$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
 3:     "contentVersion": "1.0.0.0",
 4:     "resources": [
 5:         {
 6:             "apiVersion": "2.0",
 7:             "name": "mysqlserver",
 8:             "type": "Microsoft.Sql/servers",
 9:             "location": "West US",
10:             "properties": {
11:                 "administratorLogin": "admin",
12:                 "administratorLoginPassword": "password$123"
13:             }
14:         }
15:     ]
16:}
```
In above template, line 5 to line 14 defines a SQL Database server resource. In addition to the apiVersion, name, and type properties, you also see a location property and a properties property. These two properties, along with a tags property, are defined by a Resource Provider Contract, which defines the communication protocol between Azure Resource Manager and Resource Providers. The properties list is defined and consumed by the specific Resource Provider. In this case, SQL Server Resource Provider has defined two properties (among others) that represent administrator login (administratorLogin) and administrator password (administratorPassword).
Deploy the modified template. Once the template has been successfully deployed, you should be able to observe on Azure Management Portal that the new SQL Database server has been provisioned.

In above template we put password as plain text, which is not a recommended practice. We’ll fix this when we discussparameters.

Defining a Child Resource

Next we’ll define a SQL Database on the newly defined SQL Database server. Naturally, a SQL Database belongs to a SQL Database server. In other words, a SQL Database is a child resource of a SQL Database server resource. And this is exactly how you would define it – to add a child element to the server JSON object we just defined.

Modify the template as shown in the following code list (line 14 – 28):

 1:{
 2:     "$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
 3:     "contentVersion": "1.0.0.0",
 4:     "resources": [
 5:         {
 6:             "apiVersion": "2.0",
 7:             "name": "mysqlserver",
 8:             "type": "Microsoft.Sql/servers",
 9:             "location": "West US",
10:             "properties": {
11:                 "administratorLogin": "admin",
12:                 "administratorLoginPassword": "password$123"
13:             },
14:             "resources": [
15:                 {
16:                      "apiVersion": "2.0",
17:                      "name": "mysqldatabase",
18:                      "type": "databases",
19:                       "location": "West US",
20:                      "dependsOn": [
21:                           "[concat('Microsoft.Sql/servers/', 'mysqlserver')]"
22:                      ],
23:                      "properties": {
24:                          "edition": "Web",
25:                          "maxSizeBytes": "1073741824"
26:                      }
27:                 }
28:             ]
29:         }
30:     ]
31:}

In above template, line 15 to line 27 defines the SQL Database resource to be added to the SQL Database server resource. You can see all resource definitions in Azure Templates follow a consistent format. In this new resource definition, we define a SQL Database named “mysqldatabase”, and specify it should be a Web edition with maximum size of 1G.

NOTE: You’ll need to pick a different name other than mysqldatabase as the name is already taken.

Please note the dependsOn property in line 20. The dependsOn property is used to define dependencies among resources. In this particular case, the property specifies that the SQL Database resource depends on the SQL Database server resource – this makes sense because a SQL Database server has to be provisioned first before a SQL Database can be added to it. When an Azure Template is deployed, Azure Resource Manger builds up a dependency graph based on these dependencies and ensures the resources are configured in the correct order.

NOTE: We’ll see another type of dependency by references later in this tutorial.

As an added exercise, define another child resource under the SQL Database Server resource to enable accesses to the sever from all IP addresses:

{
    "apiVersion": "2.0",
    "type": "firewallrules",
    "name": "AllowAllWindowsAzureIps",
    "location": "West US",
    "dependsOn": [
        "[concat('Microsoft.Sql/servers/', 'mysqlserver')]"
    ],
    "properties": {
        "endIpAddress": "255.255.255.255",
        "startIpAddress": "0.0.0.0"
    }
}

Using Variables

In our current version of template, the SQL Database Server name “mysqlserver” has appeared in several different places. Since it’s not a good practice to repeat string literals in your template, you can use variables to correct this.

In an Azure Template, you can define an arbitrary JSON key/value dictionary named variables. And then, you can refer to any defined values in your template by the syntax [variables(‘[key]’)]. A property value can be a simple type or a complex type. As long as you follow standard JSON syntax, you can define very complex properties if you’d like. However in this tutorial we’ll use only string properties.

Add a variables element to the template root:

…
"variables": {
    "serverName":"mysqlserver"
},
…

Once a variable is defined, you can refer it in your template code. For instance, instead

"name": "mysqlserver"

use:

"name": "[variables('serverName')]"

And instead of

"[concat('Microsoft.Sql/servers/', 'mysqlserver')]"

use:

"[concat('Microsoft.Sql/servers/', variables('serverName'))]

Using Parameters

It’s often useful to get an Azure Template parameterized. For instance, instead of fixing the SQL Database server name in the script, you can define the server name as a parameter so that the template can be used to deploy multiple instances of identically configured servers. Another use case of parameters is to separate secure information such as passwords from the template. In the following steps, we’ll define both a serverLocation parameter and anadminPassword parameter.

To define parameters, add a new parameters element to the template root. Each parameter is defined as a complex type with a type property and optional allowedValues and defaultValue properties. In current schema the supported types are: int, string, securestring and bool. The following code segment defines a string-typedserverLocation parameter and an adminPasswword parameter as a secure string.
```
…
"parameters": {
    "serverLocation": {
        "type": "string"
    } ,
    "adminPassword": {
        "type": "securestring"
    }
},
…
```
When a parameterized template is deployed, Azure reads parameter values from a separate parameter file, which is another JSON file with a list of parameter values. You can create multiple parameter files for different environments. This helps you to make sure all your environments are configured in a consistent manor.

In Visual Studio, replace the content of WebSiteSQLDatabase1.dev.json with the following code list:

{
    "$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "serverLocation": {
            "value": "West Us"
        }
    }
}

Note we are not hard-coding the adminPassword value in this file. We’ll manually enter the value during deployment.

Then, replaces all “West US” constants in your template to parameter references:
```
[parameters('serverLocation')]
```
And replace all "password$123" with
```
[parameters('adminPassword')]
```
In Solution Explorer, right-click the template and select the Deploy Template menu. On the Deploy Template dialog, you can choose which template parameter file to use. Click the Edit Parameters button and enter a value for the adminPassword parameter.
Click the Deploy button to deploy the template again. The template should be deployed correctly.

Resource Dependencies

In the “Defining a Child Resource” section you’ve seen how you can add a dependency to a resource. In this part you’ll extend the template to include a definition of a web site, which is to be deployed on Azure Websites. In this case, the web site resource depends on an Azure Websites host plan resource.

Add the following resource definitions to your template:
```
{
    "apiVersion": "2014-04-01",
    "type": "Microsoft.Web/serverFarms",
    "name": "myhostplan",
    "location": "[parameters('serverLocation')]",
    "properties": {
        "name": "myhostplan",
        "sku": "Free"
    }
},
{
    "apiVersion": "2014-04-01",
    "type": "Microsoft.Web/Sites",
    "name": "mywebsite",
    "location": "[parameters('serverLocation')]",
    "dependsOn": [
        "[concat('Microsoft.Web/serverFarms/', 'myhostplan')]"
    ],
    "properties": {
        "name": "mywebsite",
        "serverFarm": "myhostplan"
    }
}
```
NOTE: You’ll need to pick a different name other than mywebsite for your website as the name is already taken.

In the above code segment, the first resource definition defines a free-tier Azure Website host plan. This shows that in addition to the resources you own, you can also use resource definitions to create references to other PaaS and SaaS services. The second resource definition defines the web site to be provisioned on the selected host plan. And the dependency is described by the dependsOn property just like you’ve seen earlier in this tutorial.
A second type of dependencies is created by references. In a resource definition, you can refer to a property from another resource definition, creating a dependency between the two. For example, in our case we’d like to enter a connection string to the SQL Database into the web.config file of our website. And in this connection string we’ll need to refer to the fully qualified domain name (FQDN) of the database server. Obviously the website resource provider has no place to understand how to find that name, but the SQL Database resource provider should be able to provide it. And this is done by the reference keyword.

Add a new child resource that represents a web.config file to the website resource, as shown in the bolded part of the following code list:

…
{
    "apiVersion": "2014-04-01",
    "type": "Microsoft.Web/Sites",
    "name": "mywebsite",
    "location": "[parameters('serverLocation')]",
    "dependsOn": [
        "[concat('Microsoft.Web/serverFarms/', 'myhostplan')]"
    ],
    properties": {
        "name": "mywebsiteddf",
        "serverFarm": "myhostplan"
    },
    "resources": [{
        "apiVersion": "2014-04-01",
        "type": "config",
        "name": "web",
        "dependsOn": [
            "[concat('Microsoft.Web/Sites/', 'mywebsiteddf')]"
        ],
        "properties": {
            "connectionStrings": [{
                "ConnectionString": "[concat('Data Source=tcp:', reference(concat('Microsoft.Sql/servers/', variables('serverName'))).fullyQualifiedDomainName, ',1433;Initial Catalog=mytestdatabase;User Id=admin@', variables('serverName'), ';Password=', parameters('adminPassword'), ';')]",
                "Name": "DefaultConnection",
                "Type": 2
            }]
        }
    }]
}
…

Please note how the connection string uses reference([SQL Database Server resource name]) to get a reference to the SQL Database server resource, and then its fullQualifiedDomainName to get the server’s fully qualified domain name.

Deploy the template again. You should see the website provisioned with the connection string correctly set.

Conclusion

In this tutorial we walked through the steps of creating a Azure Template that defines both a website and SQL Database. And we studied how to link them together using resource dependencies. We also covered how to use variables and parameters in Azure Template and how to use different parameter files for different deployment environments.

Tuesday, 20 September 2016

Introducing Windows Azure WebJobs

I'm currently running 16 web sites on Windows Azure. I have a few Virtual Machines, but I prefer to run things using "platform as a service" where I don't have to sweat the underlying Virtual Machine. That means, while I know I can run a Virtual Machine and put "cron" jobs on it, I'm less likely to because I don't want to mess with VMs or Worker Roles.

There are a few ways to run stuff on Azure, first, there's IAAS (Infrastructure as a Service) which is VMs. Then there's Cloud Applications (Cloud Services) where you can run anything in an Azure-managed VM. It's still a VM, but you have a lot of choice and can run Worker Roles and background stuff. However, there's a lot of ceremony if you just want to run your small "job" either on a regular basis or via a trigger.

Looking at this differently, platform as a service is like having your hotel room fixed up daily, while VMs is more like managing a house yourself.

As someone who likes to torch a hotel room as much as the next person, this is why I like Azure Web Sites (PAAS). You just deploy, and it's done. The VM is invisible and the site is always up.

However, there's not yet been a good solution under web sites for doing regular jobs and batch work in the background. Now Azure Web Sites support a thing called "Azure WebJobs" to solve this problem simply.

SCALING A COMMAND LINE APPLICATION WITH AZURE WEBJOBS

When I want to do something simple - like resize some images - I'll either write a script or a small .NET application. Things do get complex though when you want to take something simple and do it n times. Scaling a command line app to the cloud often involves a lot of yak shaving.

Let's say I want to take this function that works fine at the command line and run it in the cloud at scale.

public static void SquishNewlyUploadedPNGs(Stream input, Stream output)

{

    var quantizer = new WuQuantizer();

    using (var bitmap = new Bitmap(input))

    {

        using (var quantized = quantizer.QuantizeImage(bitmap))

        {

            quantized.Save(output, ImageFormat.Png);

        }

    }

}

WebJobs aims to make developing, running, and scaling this easier. They are built into Azure Websites and run in the same VM as your Web Sites.

Here's some typical scenarios that would be great for the Windows Azure WebJobs SDK:

Image processing or other CPU-intensive work.
Queue processing.
RSS aggregation.
File maintenance, such as aggregating or cleaning up log files.
Other long-running tasks that you want to run in a background thread, such as sending emails.

WebJobs are invoked in two different ways, either they are triggered or they are continuously running. Triggered jobs happen on a schedule or when some event happens and Continuous jobs basically run a while loop.

WebJobs are deployed by copying them to the right place in the file-system (or using a designated API which will do the same). The following file types are accepted as runnable scripts that can be used as a job:

.exe - .NET assemblies compiled with the WebJobs SDK
.cmd, .bat, .exe (using windows cmd)
.sh (using bash)
.php (using php)
.py (using python)
.js (using node)

After you deploy your WebJobs from the portal, you can start and stop jobs, delete them, upload jobs as ZIP files, etc. You've got full control.

A good thing to point out, though, is that Azure WebJobs are more than just scheduled scripts, you can alsocreate WebJobs as .NET projects written in C# or whatever.

MAKING A WEBJOB OUT OF A COMMAND LINE APP WITH THE WINDOWS AZURE WEBJOBS SDK

WebJobs can effectively take some command line C# application with a function and turn it into a scalable WebJob. I spoke about this over the last few years in presentations when it was codenamed "SimpleBatch." This lets you write a simple console app to, say, resize an image, then move it up to the cloud and resize millions. Jobs can be triggered by the appearance of new items on an Azure Queue, or by new binary Blobs showing up in Azure Storage.

NOTE: You don't have to use the WebJobs SDK with the WebJobs feature of Windows Azure Web Sites. As noted earlier, the WebJobs feature enables you to upload and run any executable or script, whether or not it uses the WebJobs SDK framework.

I wanted to make a Web Job that would losslessly squish PNGs as I upload them to Azure storage. When new PNGs show up, the job should automatically run on these new PNGs. This is easy as a Command Line app using the nQuant open source library as in the code above.

Now I'll add the WebJobs SDK NuGet package (it's prerelease) and Microsoft.WindowsAzure.Jobs namespace, then add [BlobInput] and [BlobOutput] attributes, then start the JobHost() from Main. That's it.

using Microsoft.WindowsAzure.Jobs;

using nQuant;

using System.Drawing;

using System.Drawing.Imaging;

using System.IO;

namespace ConsoleApplication1

{

    class Program

    {

        static void Main(string[] args)

        {

            JobHost host = new JobHost();

            host.RunAndBlock();

        }

        public static void SquishNewlyUploadedPNGs(

            [BlobInput("input/{name}")] Stream input,

            [BlobOutput("output/{name}")] Stream output)

        {

            var quantizer = new WuQuantizer();

            using (var bitmap = new Bitmap(input))

            {

                using (var quantized = quantizer.QuantizeImage(bitmap))

                {

                    quantized.Save(output, ImageFormat.Png);

                }

            }

        }

    }

}

CONTEXT: Let's just step back and process this for a second. All I had to do was spin up the JobHost and set up a few attributes. Minimal ceremony for maximum results. My console app is now processing information from Azure blob storage without ever referencing the Azure Blob Storage API!

The function is automatically called when a new blob (in my case, a new PNG) shows up in the input container in storage and the Stream parameters are automatically
"bound" (like Model Binding) for me by the WebJobs SDK.

To deploy, I zip up my app and upload it from the WebJobs section of my existing Azure Website in the Portal.

Here it is in the Portal.

I'm setting mine to continuous, but it can also run on a detailed schedule:

I need my WebJob to be told about what Azure Storage account it's going to use, so from my Azure Web Site under the Connection Strings section I set up two strings, one for the AzureJobsRuntime (for logging) and one for AzureJobsData (what I'm accessing).

For what I'm doing they are the same. The connection strings look like this:

DefaultEndpointsProtocol=https;AccountName=hanselstorage;AccountKey=3exLzmagickey

The key here came from Manage Access Keys in my storage account, here:

In my "Hanselstorage" Storage Container I made two areas, input and output. You can name yours whatever. You can also process from Queues, etc.

Now, going back to the code, look at the parameters to the Attributes I'm using:

public static void SquishNewlyUploadedPNGs(           

   [BlobInput("input/{name}")] Stream input,            

   [BlobOutput("output/{name}")] Stream output)

There's the strings "input" and "output" pointing to specific containers in my Storage account. Again, the actual storage account (Hanselstorage) is part of the connection string. That lets you reuse WebJobs in multiple sites, just by changing the connection strings.

There is a link to get to the Azure Web Jobs Dashboard to the right of your job, but the format for the URL to access is this: https://YOURSITE.scm.azurewebsites.net/azurejobs. You'll need to enter your same credentials you've used for Azure deployment.

Once you've uploaded your job, you'll see the registered function(s) here:

I've installed the Azure SDK and can access my storage live within Visual Studio. You can also try 3rd party apps like Cloudberry Explorer. Here I've uploaded a file called scottha.png into the input container.

After a few minutes the SDK will process the new blob (Queues are faster, but blobs are checked every 10 minutes), the job will run and either succeed or fail. If your app throws an exception you'll actually see it in the Invocation Details part of the site.

Here's a successful one. You can see it worked (it squished) because of the number of input bytes and the number of output bytes.

You can see the full output of what happens in a WebJob within this Dashboard, or check the log files directly via FTP. For me, I can explore my output container in Azure Storage and download or use the now-squished images. Again, this can be used for any large job whether it be processing images, OCR, log file analysis, SQL server cleanup, whatever you can think of.

Azure WebJobs is in preview, so there will be bugs, changing documentation and updates to the SDK but the general idea is there and it's solid. I think you'll dig it.

Hook yourself to CLOUD