james

Kubernetes 1.14 brought support for running native Windows containers.

The first step to running a Windows container in your Kubernetes cluster is setting up your node. This Microsoft guide is an excellent resource to get started! Windows Server 2019 brought with it support for container networking (with Flannel), so I'm going to be using that. I'm going to assume a PowerShell interface for all Windows node and pod setup and debugging, though for some debugging it may be easier to connect to the container via a PowerShell session from an RDP instance on the node.

I'm not going to cover setting up Flannel CNI or your Linux master nodes. The Microsoft guide above is a great starting point though.

Throughout this post, I'll try to describe the areas where I had trouble and what I did to resolve it.

Update Windows

There is no easy way to update Windows in a remote PowerShell session (it requires special permissions that are not inherited from the user for some reason), so we set up a scheduled task that runs as System that is set to run once, and then restart the computer.

This could take several minutes, but we want to be on version 1903 or better for container overlay networking support.

$action = New-ScheduledTaskAction -Execute "powershell.exe" -Argument "-Command ipmo PSWindowsUpdate; Get-WUInstall -AcceptAll -Download -Install | Out-File C:\PSWindowsUpdate.log"
$trigger = New-ScheduledTaskTrigger -Once -At (Get-Date)
$principal = New-ScheduledTaskPrincipal "SYSTEM"
$settings = New-ScheduledTaskSettingsSet
$task = New-ScheduledTask -Action $action -Principal $principal -Trigger $trigger -Settings $settings

Register-ScheduledTask t1 -InputObject $task

Start-ScheduledTask t1

Restart-Computer

Install Docker

Of course, we need Docker installed

Install-Module -Name DockerMsftProvider -Repository PSGallery -Force -AcceptLicense -SkipPublisherCheck

Install-Package -Name Docker -ProviderName DockerMsftProvider

Install the Docker images

These Docker images set up the “pause” image that Kubernetes uses to set up the pod's IP address and network namespace. This is a great post on pause containers.

docker pull mcr.microsoft.com/windows/nanoserver:1809

docker tag mcr.microsoft.com/windows/nanoserver:1809 microsoft/nanoserver:latest

docker run microsoft/nanoserver:latest

Prepare for Kubernetes

The guide mentioned in the intro puts everything in a C:\k directory, and I think that is a good convention. We also need to copy over the Kubernetes certificate.

On the node:

mkdir C:\k

On your machine:

$configPath = "C:\path\to\kubernetes\config"
$session = New-PsSession -ComputerName $windowsK8sNode

Copy-Item -path $configPath -Destination C:\k\config -ToSession $session

Downloading and installing the Kubernetes binaries

We need to download kubectl, kube-proxy, and kubelet binaries. You can find the released binaries in the releases for Kubernetes on Github.

We will need 7zip to extract the contents from the .tar.gz file (as of writing, PowerShell has no means to expand this type of archive).

Note: Please verify for yourself all links and binaries before running them!

Invoke-WebRequest https://www.7-zip.org/a/7z1900-x64.msi -outFile 7zip.msi

msiexec.exe /i 7zip.msi

$env:Path += ";C:\Program Files\7-Zip\"

Now download and install the Kubernetes binaries.

Invoke-WebRequest "https://dl.k8s.io/v1.14.0/kubernetes-node-windows-amd64.tar.gz" -outFile kubernetes.tar.gz

7z x .\kubernetes.tar.gz
7z x .\kubernetes.tar

Get-ChildItem kubernetes\node\bin\*.exe |% { Copy-Item $_.FullName C:\k }

rm .\kubernetes.tar 
rm .\kubernetes.tar.gz
rm .\7zip.msi

Verify binaries installed correctly, and the node can access the cluster. Set the PATH environment variable and the KUBECONFIG environment variable so everything knows where to look.

$env:Path += ";C:\k"
[Environment]::SetEnvironmentVariable("Path", $env:Path + ";C:\k", [EnvironmentVariableTarget]::Machine)

$env:KUBECONFIG="C:\k\config"
[Environment]::SetEnvironmentVariable("KUBECONFIG", "C:\k\config", [EnvironmentVariableTarget]::User)

kubectl version

You should not see any errors.

Joining the node to the cluster (using Flannel)

Thankfully Microsoft provides a bootstrapping script that we can take advantage of.

Set-Location C:\k

Invoke-WebRequest https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/start.ps1 -OutFile start.ps1

# Make sure Docker is running
Start-Service docker

.\start.ps1 -ManagementIP <Windows Node IP> -NetworkMode <network mode>  -ClusterCIDR <Cluster CIDR> -ServiceCIDR <Service CIDR> -KubeDnsServiceIP <Kube-dns Service IP> -LogDir <Log directory>

This takes a moment or two. Afterwards, you should be able to run kubectl get nodes and see your Windows node in the cluster.

Tainting the node

To ensure that only Windows containers are scheduled on your Windows node, you can taint your node, and then apply tolerations in your pod specs.

kubectl taint nodes <node name> windows=true:NoSchedule

And then in your pod.yaml or deployment.yaml

tolerations:
  - key: "windows"
    value: "true"
    effect: "NoSchedule"

You could also use nodeSelector, but I found some Linux pods, such as kube-proxy, were sometimes assigned to the Windows node. To prevent this you can taint the node instead of trying to select specific nodes for each pod. Either way would accomplish the same goal.

Other notes and considerations

Running as a service

You can set up kubelet, kube-proxy, and flanneld to run as a Windows service, so a node failure does not require manual intervention. This worked for me:

Set-Location C:\k

Start-BitsTransfer https://nssm.cc/release/nssm-2.24.zip

Expand-Archive .\nssm-2.24.zip nssm-2.24/

Copy-Item nssm-2.24\nssm-2.24\win64\nssm.exe .

Invoke-WebRequest https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/register-svc.ps1 -outFile register-svc.ps1

.\register-svc.ps1 -NetworkMode <Network mode> -ManagementIP <Windows Node IP> -ClusterCIDR <Cluster subnet> -KubeDnsServiceIP <Kube-dns Service IP> -LogDir <Directory to place logs>

Troubleshooting

See the Microsoft guide in the intro for some more troubleshooting, here are some issues I ran into.

My Windows node is NotReady

Make sure that the kubelet process is running on the node: Get-Process -Name kubelet. If you set it up as a Windows service, make sure the service is running: Start-Service kubelet.

My Windows pod cannot communicate with the Internet

Make sure that the pod can hit the kubelet on port 10250. Check Windows firewall on the node and open this port if necessary.

Also make sure that the kube-proxy process is running: Get-Process -Name kube-proxy. If it is not, either run the start script again, or start the service.

Setting up the .NET Framework Container

In the next post, we'll go over how to set up the Windows Dockerfile to run an ASP.NET Framework application in IIS, and some of the issues that came up with that.

---