How to upload an image VHD to Windows Azure using CSUpload

Introduction

One of the exciting new possibilites in Windows Azure Infrastructure as a Service (IaaS) is the ability to create Virtual Machine images (VHD files) locally and upload them to your own image store in Windows Azure, and use them as the basis for new VMs. In this article I will demonstrate the process of uploading such an image and creating a virtual machine from it in Windows Azure.

CSUpload

There are many Cloud Storage explorers out there that can connect to Windows Azure blog storage to upload files. In this case I will use CSUpload.exe from Microsoft which is part of the Windows Azure SDK. It looks to have the best security and is specifically designed to upload images to Windows Azure. After you have installed the Windows Azure SDK the easiest way to locate CSUpload is to start the Windows Azure Command Prompt and change to the bin directory.

Disks

You can have three types of VHD files, or disks, in Windows Azure:

  • OS Images: These are generalized, read-only, base disks used for creating new virtual machines. They are either provided by Microsoft, third parties or you.
  • OS Disks: These are specialized, dedicated, writable disks that serve as the OS disk for a specific machine. If you have migrated a VM from your local virtualiztion platform to run it unmodified in Windows Azure you will use this kind of disk. Max size for an OS disk is 127 GB.
  • Data Disks: Writable disks that do not contain an operating system. They can either be migrated from your local network or created from Windows Azure. Max size for a data disk is 1 TB. The VM size determines how many data disks you can attach.

CSUpload can upload virtual disks with either the VHD or AVHD extension, meaning you can upload both parent (VHD) and child (AVHD) disks.

Setting up

The first thing we need to do is create a connection string for CSUpload to user for connecting to your Windows Azure subscription. You do this with the CSUpload Set-Connection command. (As you can see CSUpload borrows from the page of PowerShell with its verb-noun command structure. Hopefully this means that it will be replaced by native PowerShell cmdlets in the near future.) CSUpload Set-Connection takes exactly one argument which is a string in quotes consisting of your Windows Azure Subscription ID, Certificate Thumbprint and Service Management Endpoint. These are most easily obtained by running the Get-AzureSubscription cmdlet from a PowerShell prompt where the Windows Azure module is loaded. Here is an example.The strings marked in red are the ones you need to use in CSUpload:

image

Here’s how the CSUpload command would look (wrapped for readability):

csupload.exe set-connection
"SubscriptionId=3cf46281-f639-44bc-a338-11697697bb2a
;CertificateThumbprint=C525C2AA4B587506F851A7EF5456068270ECC969
;ServiceManagementEndpoint=https://management.core.windows.net"

Windows(R) Azure(TM) Upload Tool version 1.7.0.0
for Microsoft(R) .NET Framework 3.5
Copyright (c) Microsoft Corporation. All rights reserved.

Updated the default connection string.

 

Once you have configured this, CSUpload will remember it until you do something to change your subscription, say delete your management certificate etc. You could also specify these parameters each time you run one of the other CSUpload commands, but that gets old real quick. Better to define them once.

The upload

To upload a disk CSUpload requires four mandatory parameters;

  • Destination: the storage account and path where the disk will be stored in Windows Azure. This is consists of two parts; the URL of the storage account, which you can get either from the portal or the cmdlet Get-AzureStorageAccount, the path where you want to put the VHD in that storage account, and the name of the VHD, including the .vhd extension.
  • Label: The disk’s label, which is displayed when you create new VMs in Windows Azure.
  • LiteralPath: The location of the VHD/AVHD file to upload, this is on your local disk.
  • OS: If the disk contains an operating system, specify either Windows or Linux.

Here is an example (wrapped for readability):

csupload.exe Add-PersistentVMImage
-Destination
http://mdsstgacc01.blob.core.windows.net/myimages/base_win7sp1x64.vhd
-Label "Windows 7 SP1"
-OS Windows
-LiteralPath "d:\Hyper-V\Virtual Hard Disks\base_win7sp1x64.vhd" –Name basewin7sp1x64.vhd

The command above produces the following output:

Windows(R) Azure(TM) Upload Tool version 1.7.0.0
for Microsoft(R) .NET Framework 3.5
Copyright (c) Microsoft Corporation. All rights reserved.

Using the saved connection string…
MD5 hash is being calculated for the file ‘d:\Hyper-V\Virtual Hard Disks\base_win7sp1x64.vhd’.
MD5 hash calculation is completed.ning Time: 00:00:00; Throughput: 2694,3Mbps
Elapsed time for the operation: 00:00:15
Creating new page blob of size 5368709632…
Elapsed time for upload: 00:00:00
Registering the image ‘basewin7sp1x64.vhd’ with repository.
Image ‘basewin7sp1x64.vhd’ is registered successfully.

Let’s look at what happens…

1. MD5 Hash

CSUpload starts by generating an MD5 hash of your file to gurantee that it has been transferred correctly.CSUpload will report its progress as a percentage and display the speed at which it reads data from you local disk. This number is given in Mbps. No actual transfer of data has started yet.

MD5 hash is being calculated for the file ‘d:\Hyper-V\Virtual Hard Disks\base_win7sp1x64.vhd’.
MD5 hash calculation is completed.ning Time: 00:00:00; Throughput: 2694,3Mbps
Elapsed time for the operation: 00:00:15

2. Allocate blob

Once the MD5 calculation is finished CSUpload will allocate a blob of the same size as the disk in your storage account.

Creating new page blob of size 5368709632…
Elapsed time for upload: 00:00:00

3. Transfer data

Now CSUpload will transfer the contents of the disk to Windows Azure. Note that only blocks in the VHD that actually contain data will be transferred, empty blocks are skipped. If you have a large disk you will see CSUpload use some time maping which blocks contain data.

Uploading: 2,2% complete; Remaining Time: 00:10:29; Throughput: 18,9Mbps
Elapsed time for upload: 00:04:22

4. Register in repository

We are using the CSUpload command Add-PersistentVMImage, which means we are uploading something that can be the basis of new VMs in Windows Azure IaaS. We need to tell Widnows Azure that it is dealing with an image so CSUpload will register the image in your subscriptions image repository. This means it will be visible both under Images in the Virtual Machine node and in your blob storage account.

Registering the image ‘basewin7sp1x64.vhd’ with repository.
Image ‘basewin7sp1x64.vhd’ is registered successfully.

image
Image in the image repository.

image
Image in the storage account.

Notes

  • I have a 50 Mbps fiber optic connection at home, and I was able to max it out while uploading the VHD to Windows Azure. It seems there is no upload limit into the datacenter when performing this kind of operation. VM sizes set a cap on bandwidth, but since the VHD is not yet associated with any VM it seems there is no limit. Further investigation required…
  • The VHD file must have a size that is a mutiple of 1024 bytes. Or else you will get this error:
    The VHD https://mdsstgacc01.blob.core.windows.net/myimages/base_win7sp1x64.vhd has an unsupported virtual size of 500000
    0000 bytes.  The size must be a whole number (in MBs).
  • For you so successfully connect to a new VM based on your image, any image you upload must have Remote Desktop enabled and allowed through the host based firewall.
  • The local Administrator account must be enabled and must have a password set. Later versions of Windows do not allow logon through Remote Desktop with a blank password.
  • You should use HTTPS when specifying you blob storage. HTTP could leak data during transfer, which CSUpload will warn you about:

    Use of http is insecure, can leak data to untrusted viewers and make your content vulnerable to data tampering by a malicious third party.

UPDATE: As of December 2012 a new cmdlet has been included in the Windows Azure PowerShell module; Add-AzureVHD. It performs the same functions as CSUpload with regards to uploading images and disks. Get the latest PS module here:

https://github.com/WindowsAzure/azure-sdk-tools/downloads

More free stuff in Windows Azure Active Directory

Microsoft announced today that they will offer two features in Windows Azure Active Directory free of charge; Access Control and Core Directory and Authentication.

  • Access Control provides centralized authentication and authorization by integrating with consumer identity providers such as Facebook or by using an on-premises deployment of the Windows Server Active Directory service. With Access Control, you can create a single application that can allow users to sign in with both their organizational credentials stored in Windows Azure Active Directory or Windows Server Active Directory, or by using popular consumer identity services like Microsoft Account, Facebook, Google, or Twitter. Historically, Access Control has been priced by number of transactions, but now it is available at no charge.
  • Core Directory and Authentication enables capabilities such as single sign-on, user and group management, directory synchronization, and directory federation. These capabilities are currently free of charge in the Windows Azure Active Directory Developer Preview and will remain free of charge after Windows Azure Active Directory reaches general availability.

Good times!

Windows Azure Architecture Guides released!

This just in: “Microsoft releases Windows Azure Architecture Guides! ITPros go wild!” Read all about it here:

http://www.microsoft.com/en-us/download/details.aspx?id=35764

A 9 MB Word-document; you gotta love it!

Morgan

How to connect your on-premise network to Windows Azure using Windows Server as a VPN gateway

Introduction

Together with the launch of Windows Azure Infrastructure as a Service (IaaS) this summer, Microsoft also introduced a way for customers to connect their on-premise networks with Windows Azure using site-to-site VPN.

image

The service responsibel for this feature is called Windows Azure Gateway. It uses IPSec to establish a site-to-site VPN tunnel between your network and your networks in Windows Azure. Effectively adding a second site to your network. Currently only Cisco and Juniper devices are officially supported as your local part of the tunnel. However since Windows Azure Gateway is using standard IPSec site-to-site tunneling you could theoretically use any device supporting the requirements. One such scenario using Windows Server 2008 R2 as our on-premise router, is the purpose of this port. (If you’re wondering why I’m not using Windows Server 2012, the answer is simply that it does not support the requirements. Specifically Windows Server 2012 does not do NAT-T like Windows Server 2008 R2 does.)

Requirements

The Windows Azure Gateway documentation lists the following requirements for the on-premise VPN device:

  • VPN device must have a public facing IPv4 address
  • VPN device must support IKEv1
  • Establish IPsec Security Associations in Tunnel mode
  • VPN device must support NAT-T
  • VPN device must support AES 128-bit encryption function, SHA-1 hashing function, and Diffie-Hellman Perfect Forward Secrecy in "Group 2" mode
  • VPN device must fragment packets before encapsulating with the VPN headers

Fortunately for us Windows Server 2008 R2 supports all of these! So let’s set it up.

Before you can configure your local device you have to perform these steps in the Windows Azure Management Portal:

  • Create one or more virtual networks (WAVN) in Windows Azure
    These will host you Windows Azure VMs and be your LANs in Windows Azure.
  • Define a local netwok in Windows Azure
    Configure this network with all the subnets that you run in your on-premise network, as well as the public IP address of you VPN device.
  • Add at least one DNS server to Windows Azure
    These can be any DNS server; on-premise, in Windows Azure and public DNS. All the servers you add will be assigned with Windows Azure DHCP to your VMs.
  • Set aside one subnet within the networks you created in Windows Azure to be the link-network
    This network represents the link between you and Windows Azure. It must be within the boundary of the networks you created in Windows Azure in the previous step.
  • Enter the public IP of your VPN device
    The VPN device cannot be behind a NAT, not even a 1:1 NAT with public IPs.
  • Start the Windows Azure Gateway

These are high-level steps. The Windows Azure documentation goes into great detail about how to configure your cloud networks etc. This post focuses on using Windows Server as you local VPN device so I will not repeat the specific steps here. Instead I refer you to the documentation:

After completing the setup in Windows Azure you are ready to configure your local device.

The Windows Server 2008 R2 machine you will be using as your VPN device must have hotfix 2523881 installed. This patch enables the old Windows Server 2003 mode of NAT-Traversal (NAT-T) which is required by Windows Azure Gateway. If you do not have this hotfix installed, you will receive traffic into your network but return traffic will not work. Here is the link to the hotfix:

The Local Network

This is the local network topology in my test environment:

image

Our task here is to connect our on-premise network with our Windows Azure networks and then promote a server in Windows Azure to a domain controller for our Active Directory domain.

Routing

Your local VPN server does not need to be the default gateway for your local network, but if it is, it will make your setup easier. Suffice it to say that you need to work out the routing requirements in your environment. In this example I assume that the VPN server is also your local default gateway. Your VPN server should not have a default gateway IP set on the NIC connected to the local network (LAN). If you require custom routing use RRAS, the route command or NETSH to set up your routes.

The VPN server

Your local Windows Server machine needs at least two NICs, one connected to your local network and one to the public Internet. The server does not need to be joined to your domain. I highly recomment you keep the Windows Firewall enabled on the VPN server. Having a server directly connected to the Internet without a firewall is not a good idea.

High level setup steps

  • Document the public IP of your VPN server and the public IP of your Windows Azure Gateway endpoint, as well as your Windows Azure networks and local networks. You will need these during setup.
  • Find you IPSec encryption key from the Windows Azure portal.
  • Enable routing
  • Configue IPSec tunnel
  • Verify connectivity
  • Add VM in Windows Azure and promote to domain controller.

Retreive the IPSec encryption key

Log on to the Windows Azure portal and select Networks. Click the network your are connectin your on-site network to and select View Key (you will find this at the bottom of the screen):

image

Copy the displayed key:

image

Enable routing

Install the Routing and Remote Access (RRAS) Role Service which is part of the Network Policy Server and Access Services role. You will need to select both Remote Access Service and Routing, one cannot be installed without the other. You can do this either through Server Manager or PowerShell.

image

The PowerShell command is:

Add-WindowsFeature NPAS-RRAS-Services –IncludeAllSubFeature

Enable and configure Routing and Remote Access for LAN routing only. Right click Routing and Remote Access and select Configure and Enable Routing and Remote Access:

image

Select Custom Configuration and the select LAN Routing:

image

What this step does is turn Windows into an IPv4/IPv6 router. It simply tells it to start forwarding IP datagrams. Unless you have special routing requirements in your environment you are finished with configuring RRAS.

Configure the IPSec tunnel

In Windows Server 2008 and newer IPSec settings have been merged into the Windows Firewall.

1. Open Windows Firewall with Advanced Security and select Connection Security Rules:

image

2. Right click and select New Rule. On the Rule Type page, select Tunnel:

image

3. On the Tunnel Type screen, leave the default Custom configuration and No for IPSec exemptions selected and click Next:

image

4. On the Requirements screen leave the default: Require authentication for inbound and outbond connections selected and click Next:

image

5. Next, on the Tunnel Endpoints screen, configure the tunnel endpoints and networks. (You will have to scroll down to configure the networks for Endpoint 2):

image
NOTE: It is extremely important that the networks you define here match your local network configuration in Windows Azure or your traffic will not be routed.

image

6. On the Authentication Methods screen, select Advanced and then press the Customize button:

image

7. In the Customize Advanced Authentication dialogue add a Preshared key authentication for the First Authentication Methods:

image

8. On the Profile screen select the Firewall Profiles for which this rule will apply. Usually it will be all three:

image

9. At the Name screen, give the rule an appropriate name and description:

image

10. Windows Azure Gateway requires that you change the TCP Maximum Segment Size (MSS) to aviod fragmentation. You do this with NETSH on your external (Internet facing) interface. First list your interfaces:

netsh interface ipv4 show subinterfaces

In the Interface column you should recognize your interface names. Now change the TCP MSS value:

netsh interface ipv4 set subinterface “<name of interface>” mtu=1350 store=persistent

Run the first NETSH command again to verify the change.

11. Configure the IPSec Quick Mode key lifetimes. Windows Azure Gateway uses a Quick Mode (Phase 2) key lifetime of 1 hour (3600 seconds) or 100 GB of traffic, whichever happens first. The Phase 1 key lifetime is 8 hours, which is the default in Windows Server 2008 R2 so there is no need to change that.

Right click the Windows Firewall with Advanced Security node at the top of the Windows Firewall console, and select Properties. Then select the IPSec tab, and press the Customize button:

image

Next select the Advanced radio button under Data protection (Quick Mode) and press the Customize button. Under Data integrity and encryption select the entry called ESP/SHA1/AES-CBC 128 etc. and press the Edit button:

image

Under Key lifetimes the timeout value in minutes is already set correctly to 60 minutes (3600 seconds) so we only need to configure the KB timeout. Set it to 102 400 000 KB (100 GB).

image

Exit out of all the boxes by pressing OK.

NOTE: These are global settings which affect all connection security rules on the server. If you want to specify these settings only on the connection security rule that pertains to Windows Azure, use NETSH. Unfortunately you cannot configur connection security rules specific main mode or quick mode settings in the GUI. Also you cannot use NETSH to configure global quick mode settings, only main mode settings. The logic behind this escapes me…. If you do decide to configure rule specific quick mode settings with NETSH, the GUI will inform you that your rule “…contains properties that are not supported through this interface”. That said I would actually recommend using rule specific quick mode settings because that way you won’t have to change the computer defaults which could potentially cause problems for other rules. Although not needed by Windows Azure Gateway, because the default settings match the required settings, you can also configure specific main mode rules that match e.g. endpoints, using NETSH. More about the diffrences between the Advanced Firewall GUI and NETSH here. Also have a look at the scripts section at the end of the post.

12. Verify that the IPSec tunnel has been created using the Monitoring node of the Windows Firewall with Advanced Security console. Under Security Associations you should have an association under both the Main Mode and Quick Mode nodes:

image

image

If you do not see any security associations, try to ping an address in one of your Windows Azure networks. This should establish the tunnel.

13. Verify connectivity in the Windows Azure portal:

image

Add Windows Azure Virtual Machines

Now you can add Windows Azure VMs to your Windows Azure networks. These machines will receive IP addresses from the Windows Azure DHCP service. The addresses will be leased to the machine for its lifetime so it will be the same as having a static IP. Windows Azure DHCP will also configure the servers with the DNS servers you have defined in Windows Azure. These can be both on premise and in the cloud. The default gateways for the machines will be set to the first address on the subet that the machine is connected to.

Verify connectivity

After the first Windows Azure VM is online (and its firewall opened) you should be able to send traffic across the VPN. In my case I can now ping between my Windows Azure VM and on-premise machines:

image

Now I can add the make the Windows Azure VM a domain controller:

image

Scripts

Since network devices like routers and switches are usually configured using scripts, here are the NETSH commands to configure Windows Server as a VPN device from the CLI:

  1. Enable IPv4 routing:
    You don’t really need RRAS installed to make Windows Server route IPv4 traffic. You can configure the same functionality directly with NETSH. This removes the requirement for RRAS to be installed. To configure routing you must first find either the interface names or indices of your network interfaces. In this case mine are 12 and 14. 12 is the External interface connected to the Internet and 14 is the Internal interface connected to the LAN. Routing, or IP datagram forwarding, must be configured on both. Use NETSH:
    netsh interface ipv4 set interface “12” forwarding=enable netsh interface ipv4 set interface “14” forwarding=enable
  2. Create connection security rule with rule specific quick mode settings:
    netsh advfirewall consec add rule name="Windows Azure" endpoint1="192.168.0.0/16" endpoint2="10.1.0.0/16" action="requireinrequireout" description="Site-to-site VPN for Windows Azure" mode="tunnel" profile="any" type="static" localtunnelendpoint="80.212.96.194" remotetunnelendpoint="168.63.16.208" protocol="any" auth1="computerpsk" auth1psk="wL8fC…" qmsecmethods="ESP:SHA1-AES128+60min+102400000kb"
  3. Create main mode rule that matches traffic between the local network and Windows Azure. This rule will then be used by the connection security rule. This is a redundant step since the default settings of Windows Server 2008 R2 match the requirements of Windows Azure. This is just to show how it’s done…
    netsh advfirewall mainmode add rule name="Windows Azure IPSec Main Mode Settings" mmsecmethods="dhgroup2:aes128-sha1" mmkeylifetime="480min,0sess" description="Main mode settings compatible with Windows Azure Gateway" endpoint1="192.168.0.0/16" endpoint2="10.1.0.0/16"
  4. Configure the TCPMSS size:
    netsh interface ipv4 set subinterface “<name of interface>” mtu=1350 store=persistent

So if you combine all the commands in a nice cmd file you have something resembling a router configuration script.

More info on NETSH syntax is available here:

Notes

  • IPv6 is currently not supported by Windows Azure so you will have no IPv6 native connectivity. Maybe you can make some of the transition technologies work. I have not tested that.
  • The VPN Server itself will not be able to communicate with virtual machines in Windows Azure. Only computers behind the VPN server will have communication.
  • I was able to measure a transfer speed of about 9 Mbps between my local network and a Small Instance VM running in Windows Azure. I used IPERF to test.
  • I have experienced some instability with this solutions but I am not sure if this has to do with Windows Server or Windows Azure. Leave a comment with your experiences.
  • The Windows Firewall protects your VPN Server on the Internet. make sure that it is configured correctly. May I also suggest to remove any unnecessary bindings on the Internet facing NIC. Also remember that the VPN traffic is not inspected by the firewall since it is inside the tunnel.
  • The configuration scripts available for supported gateway devices, currently Cisco and Juniper, are quite useful to understand the settings needed to configure your VPN server. These are available for download through the Windows Azure Managmenet Portal, under Networks.
  • Windows Azure PowerShell can also be used to manage and set up your networks and gateway.

Windows Azure Camp in Norway

For all of you living in Norway who are interested in learning more about Windows Azure; you can sign up for one of the four free Windows Azure Training Camps I will give at the Microsoft offices in Oslo. We will do one training camp on the first Wednesday of each month for the rest of 2012. The camps will include both aa instructor led presentation and hands on labs (so remember to bring your own device!). The agenda looks like this:

Presentaion: Windows Azure Virtual Machines
Comprehensive presentation that introduces the technical concepts and business value of Windows Azure Virtual Machines and Virtual Networks
Hands-on Lab: Introduction to Windows Azure Virtual Machines (Windows)
In this hands-on lab you will learn how to deploy a simple ASP.NET MVC3 Web application to a Web server hosted in Windows Azure, using SQL Server and configuring load balancing.

Presentation: Deploying Active Directory in Windows Azure
Presentation Understanding how and when to deploy Active Directory within Windows Azure Virtual Machines
Hands-on Lab: Deploying Active Directory in Windows Azure (PowerShell)
In this Hands-on lab you will walk through the steps necessary to deploy a stand-alone domain in the cloud using Windows Azure Virtual Machines and Virtual Networks using PowerShell.


Presentation: Automating Virtual Machine Management with PowerShell 
Introduction to using the Windows Azure PowerShell Cmdlets to manage and automate virtual machines and virtual networks.
Hands-on Lab: Managing Virtual Machines with the Windows Azure PowerShell Cmdlets
In this hands-on lab you will understand the capabilities of automating the deployment and management of virtual machines in Windows Azure.

Register for the training camps here: https://msevents.microsoft.com/CUI/EventDetail.aspx?culture=en-US&EventID=1032524002

Hope to see you there!