james

Serverless is an architecture pattern that allows the user to not worry about servers. Of course servers still exist in a data center somewhere, but they do not concern the customer. These are many of the benefits to using this pattern.

Abstractions

“All problems in computer science can be solved by another level of indirection... Except for the problem of too many layers of indirection.” — David Wheeler

Serverless architecture abstracts the hardware from the software. In the same way that an operating system can provide a framework and environment without the user having to know where to put things in memory, the cloud allows you to write software without having to worry about how it is actually running.

Abstractions are useful because they allow us to think at a higher level. We can think in the solution space, rather than in unnecessary details like config file management and patching regimens. Many people have wasted many hours tracing down that config setting in their load balancer, and dealing with rolling out patches gradually so as to not interrupt service.

These things, and many more like it, must still happen, but they are done by experts, or at least people dedicated to doing that as their job. In Domain Driven Design we would classify this as a generic sub-domain; something that is required, but not core to the business goals. By abstracting the generic management stuff away, you are left with only the problem core to the business, which allows faster iteration on the solution that will generate income.

There are a lot of benefits to abstracting the server infrastructure out of your stack. If you are able to leverage the cloud completely to reach your business goal or solve your problem, you can significantly reduce cognitive overhead and maintenance, and increase development velocity.

Cognitive Overhead

When running a traditional infrastructure, there is a significant need to have a decently accurate mental model. Most production infrastructures are complex enough that it is nearly impossible to know every bit of it in sufficient detail. Some of this doesn't go away with serverless. Here are some of the benefits, in my mind:

In a traditional infrastructure you need to be able to predict and provision resources ahead of time so you don't go down. Serverless infrastructure is immutable by default and since you don't provision servers, and you don't have configuration files to manage, there are fewer moving parts. A traditional infrastructure could be set up in an immutable fashion, but you must go out of your way to set that up and to maintain it. Instead of thinking how many servers you need, you think in terms of what resources and services help you solve your problem.

A large part of the cognitive overhead of running a traditional infrastructure is maintenance. You need to be able to keep your servers up, running, and patched.

Maintenance

Maintenance is a big part of any robust infrastructure. With serverless you can minimize it. There are no servers to keep patched and updated, there are no application configuration files you need to store in puppet or chef.

There is still maintenance you need to perform for serverless. You may need to optimize functions and queries, manage log tracing and analytics, and prune unused resources occasionally.

Even with serverless, there is still the issue of the cloud provider messing up. Google and Amazon engineers are not infallible, and have often made configuration errors that resulted in downtime. Examples: Google storage downtime AWS S3 downtime

Most cloud resources have a lot of 9's of reliability, however. AWS S3 for instance is has 99.99% availability and 99.999999999% durability over a year. This means that there should be no more than 52 minutes of downtime, and if you store 10,000,000 objects you could expect to lose an object every 10,000 years. The experts at the cloud provider work hard to keep the backend up, so you don't have to. Your company doesn't have to have as much staff on hand to keep things running.

Scale to Zero

Scale to Zero is the concept of having no provisioned resources when you have no traffic. This has a large benefit of saving a ton of money, and not being wasteful. However, a downside is that there can be a lag between when a request comes in and resources are available to handle the request.

In a traditional infrastructure you would have to provision enough resources to handle the peak load. This means that (taking the 80-20 rule for sake of argument), you are not using your resources to their capacity 80% of the time. Yet you must pay to run these servers. Electricity must be used, coal burned, turbines spun, and the only output is unnecessary heat.

In aggregate at cloud scale, serverless resources can be provisioned dynamically and much more dense to use nearly 100% of the available power. Of course in reality the datacenter must keep a lot of hardware in standby ready to take up the load, however, this is minimized in the case of serverless.

Here is a very detailed graph of this concept, as an example. pretty graph showing provisioned resources as a constant line, while serverless resources follow the curve of user traffic

I don't think everyone should rush to implement serverless architectures to try to save the planet – there are better reasons to go serverless – but the optimization in cost should be considered. Because you only pay for what you use, your cost structure can be related more directly to your usage patterns. If you are just starting out and have few users, your costs will be low. Your costs scale with you, and can be factored in to the pricing model, ideally.

Serverless scales to your credit card, and this allows you to focus on what really matters to the business.

Don't reinvent the wheel, focus on business value

There is a lot of technology involved in large enterprise infrastructure projects. A lot of networking, load balancing, firewalls, general logic, and things otherwise unrelated to generating income for your business, or unrelated to the problem you are solving.

So many of these parts of the infrastructure are not necessary to provision and manage with a serverless architecture. They have been done for you, there is no need to waste time and reinvent the wheel. This immediately gives businesses that adopt serverless from the start a leg-up. The engineers can focus on core value.

With a good software architecture and corresponding business architecture (Conway's Law), your development velocity should be much higher. There are fewer barriers between writing the software and delivering it to your customers.

Speaking of Conway's Law, a serverless pattern leans heavily into microservices by default. To implement a well architected solution requires a business organization that is complementary. This can have many benefits such as faster iterations with smaller teams, and fewer bugs since changesets are smaller.

Future Impact

There is no current standard for serverless computing. The International Data Center Authority mentions it in their AE360 paper (the compute layer), but this is not a robust standard.Because there is no standard, you have to decide between the three major cloud providers, or roll your own. There is the Open FaaS project that can help with functions, and you could pair this together other SaaS products, but you would need to write a lot of integration code.

There is a real concern of vendor lock-in, which really limits the ability to migrate, and introduces a bit of unpredictability. If serverless architecture is going to be the future, I expect more standardization around common patterns will be necessary. Another huge benefit to standardization is consolidation of tooling. Terraform providers, and each cloud's specific tools are great, but they lack features and are an obstacle to deploying infrastructure as code. This is also how cloud providers could benefit from standardization.

Things I hope to see standardized:

  • Compute
  • API
  • Storage
  • Caching
  • CDN
  • Messaging/Eventing
  • Monitoring
  • Analytics

Outside of standardization, I see more layers of abstraction being built up. For instance, in AWS, why should the user decide between RDS Aurora, DynamoDB, or Redshift? If there was a standard for serverless storage, a user could identify their access patterns and requirements, and have the proper infrastructure provided automatically.

Another method of abstraction would be to remove the notion of regions. Infrastructure could be moved around seamlessly to serve patterns of traffic automatically. Automatically provisioning instances of compute and replicating data to storage that is closer to the user.

Regardless of standardization or further abstractions, serverless is set to have a huge impact on computing. The industry is still growing rapdily, and more and more companies are moving to the cloud and going serverless.

Arguments Against

There are a lot of reasons why you should not go serverless. Vendor lock in, cost, knowledge, Capex vs Opex, and it may just not provide the functionality that is required for your problem or business.

These are all valid concerns and need to be considered when planning your next architecture or migration.

Vendor Lock In

This is the fear of putting all your eggs in one basket. Now that all of your infrastructure is in AWS, what happens when AWS raises their prices or removes a product that you are depending on? How quickly can you migrate away from your 14 TB RDS instance?

This fear is alleviated somewhat with more cloud providers. Competition provides a bit of incentive for prices to remain low and predictable, because it would be easy to undercut for similar or equivalent products.

With IaC, it becomes easier to replicate infrastructure in a competing cloud environment while still running in the existing one. Migrating your data is a trickier issue, but with careful planning and services like AWS Snowball, it is definitely possible.

There are also hybrid solutions. Perhaps you balance your infrastructure across multiple cloud environments. This is tricky to set up, and maintenance heavy, but if vendor lock in is a major concern, this is one alternative.

Cost

Cost can be make it or break it for a business just starting out, or one with tight margins. If you have a new business with unknown usage patterns, however, serverless can be a perfect fit. Since you can scale to zero, you can save a lot of money when your customers aren't using your platform.

If you are a business with tight margins, you probably have well-known and predictable access patterns. This is one case where serverless may not be the solution. If you can provision ahead of time the exact amount of resources required, then you can save a lot of money.

However, this allows no room for growth, at least not rapidly. You could find yourself in a position where you just can't handle the load the users are putting on your system. In that case, automatically scaling with cloud infrastructure could keep your business alive and making money.

A hybrid solution in that case may be the perfect fit. You have provisioned resources for the base load, but can burst into the cloud when necessary to handle surges of traffic. This requires some careful planning, but is definitely possible, though this strays away from serverless a bit.

Knowledge

Knowledge can easily be a barrier to entry to new technology. New things take time and persistence to learn. You may be hesitant to jump into serverless because there is just too much going on, and things are working just fine right now, thank you very much.

Fortunately, there are a lot of resources for learning this technology. From acloud.guru to linuxacademy, and all of the certificate training courses, you have not only the teaching material, but also the accreditation to show that you learned it.

One of the huge benefits to the computer software industry is that anyone can learn it if they put the time and effort it. It is a great equalizer. But it does require the effort.

Capex vs Opex

Your company may be unwilling to migrate to serverless, or the cloud in general, because it is not a capital expense. The company does not own anything for equity purposes, they are merely renting hardware. I am not a financial advisor or an accountant, but this seems like a silly argument. Any money saved by clever accounting for tax purposes could be overshadowed by the amount of money saved by having a more efficient infrastructure. Your company makes its money from software, and that software needs to run on something.

You can pay people money to maintain and build that infrastructure, or you could pay a third party to do it for you. There is a good chance that your staff of infrastructure, network, and software engineers to build and maintain that will be more expensive.

Missing functionality

If you have a niche business or scientific experiment, you may not find the features you require in the cloud. There may be special requirements for privacy, security, or strange computations that just aren't there. In some cases this is absolutely true.

For instance, I don't see a quantum computer class of EC2 instances coming to AWS any time soon. In a lot of other cases, however, reframing the problem to use the provided tools is possible, and may even lead to new, more efficient solutions.

Perhaps you do not need to calculate everything at once in a large simulation, and an eventually consistent event-driven architecture would get the job done as well. This is more of a case-by-case thing, but it is worth considering alternative approaches because they could turn out to be more cost effective and efficient.

Conclusion

Serverless is a great form of computing, and it is here to stay. The industry is still growing rapidly, and for good reason. Companies can save a lot of money and time by using only the resources they need, and cutting out self-hosted infrastructure solutions.

Money can be saved not only in total run time costs, but also in manpower. Staffing requirements can be lessened, or people can be moved onto solving the business goals that actually generate income.

Serverless is a great abstraction on infrastructure in general. It provides the tools to think in the solution space, instead of all the steps required to set it up.

An integral part of running a Kubernetes cluster in production is security, including how human users authenticate with the API server (the public interface of the cluster). A properly configured cluster should allow only those with proper credentials access. And those with access should only have the ability to interact with the objects necessary to complete their jobs. This is the principle of least privilege. I may go into RBAC in Kubernetes later, if I feel like I have anything interesting to say on the topic. For now, let's see how authentication can work with OIDC.

The Kubernetes documentation has a very good article on Authenticating which I suggest you read for more in-depth knowledge on this topic. I am focusing on the OpenID Connect Tokens section. I am going to assume some basic understanding of OAuth2 and OIDC flows. If you are not familiar, here is a good illustrated guide I saw recently come up on Hacker News.

Configuring the Kubernetes API server for OIDC

The first step in this process is configuring the API server for OpenID Connect. There are several arguments that need to be passed into the server on startup to get this all wired in. You can see the full list here, but I am going to mention one:

—oidc-username-claim

This argument defaults to the sub value, which may work for you. I changed this to the email field for it to be a little more user friendly when assigning UserRoleBindings later on (it makes it clear who is gaining access to what, especially if you are using infrastructure as code practices).

Once you have configured the client id, URL, and CA for your OIDC provider (idp), you can get a token and attempt to connect.

Getting the token from your identity provider

To retrieve the token from your identity provider, you will need a few things:

  • client id
  • client secret
  • grant type
  • scopes requested
  • username
  • password

Let's do this with some PowerShell

$creds = Get-Credential -Message "Provide your $identityServerUrl credentials."

This will generate a prompt for the user to enter in their username and password, and stores the password in a System.Security.SecureString object. This ensures that the data is removed from memory properly when no longer needed (of course a clever user who has access to the system would probably be able to get the data if they really wanted, but it indicates that this is sensitive information).

Now let's create the body for the request

$body = @{
  "scope" = "kubernetes email_only offline_access";
  "grant_type" = "password";
  "client_id" = "$clientId";
  "client_secret" = $secret;
  "username" = $creds.UserName;
  "password" = $($creds.GetNetworkCredential().Password);
}

You will have to fill in the details with your own idp. You can also use different scopes if you wish. The kubernetes scope was one I added to handle specific information for Kubernetes based authentication.

$identityServerTokenEndpoint = "$identityServerUrl/identity/connect/token"
$tokenString = Invoke-RestMethod -Uri $identityServerTokenEndpoint -Method Post -Body $body

Now we can call the token endpoint to get our token from the identity provider. The response should have in it an access token and a refresh token (it may have an id_token instead, depending on identity provider). We can use the access token as the identity token in our Kubernetes configuration file. We can set our .kube config using kubectl, and then use the new context to authenticate with our cluster:

$accessToken = $tokenString.access_token
$refreshToken = $tokenString.refresh_token

$clusterName = "kubernetes-idp"
$contextName = "$clusterName"

& kubectl config set-cluster $clusterName --server=$k8sServerUrl --certificate-authority=$k8sCaCrt

& kubectl config set-context $contextName --cluster=$clusterName --user="$($creds.UserName)"

& kubectl config set-credentials "$($creds.UserName)" --auth-provider=oidc --auth-provider-arg=idp-issuer-url="$identityServerUrl/identity" --auth-provider-arg=client-id=$clientId --auth-provider-arg=client-secret=$secret --auth-provider-arg=refresh-token=$refreshToken --auth-provider-arg=idp-certificate-authority=$identityCaCrt --auth-provider-arg=id-token=$accessToken

& kubectl config use-context $contextName

Where $identityCaCrt is the identity provider's CA for its public domain certificate. The other variables should be self-explanatory. Just fill in the details of your cluster.

The token returned by the identity provider is a JWT. One downside with JWTs is that there is no mechanism to invalidate them directly (because the client controls the certificate, it is an asynchronous process by design). We can alleviate concerns by setting a short TTL on these tokens, but this leads to another issue: it can be annoying to keep getting these tokens. This is what the refresh token is for.

Setting the refresh token should help with generating valid credentials while this token is being used. However, if you go an extended period without using your token, it will be invalid the next time you try to use it. One way to help with this issue is a scheduled task or cronjob to continually generate these credentials and update your Kubernetes config file.

Refresh the token

To refresh the token, we make a refresh token request to the identity provider, using the refresh token we got initially.

$body = @{
  "grant_type" = "refresh_token";
  "client_id" = "$clientId";
  "client_secret" = $secret;
  "refresh_token" = $refreshToken;
}

$identityServerTokenEndpoint = "$identityServerUrl/identity/connect/token"
$tokenString = Invoke-RestMethod -Uri $identityServerTokenEndpoint -Method Post -Body $body

As an example to register a scheduled task on Windows that runs every 50 minutes (assuming the TTL on the token is 60 minutes):

$trigger = New-JobTrigger -Once -At ((Get-Date).AddMinutes(50)) -RepetitionInterval (New-TimeSpan -Minute 50) -RepeatIndefinitely
Register-ScheduledJob -Name "K8s-OIDC-Refresh" -Trigger $trigger -ScriptBlock {
    param(
        [string]$clientId, 
        [string]$secret,
        [string]$refreshToken,
        [string]$identityServerUrl,
        [string]$identityCaCrt,
        [pscredential]$creds
    )

    $body = @{
        "grant_type" = "refresh_token";
        "client_id" = "$clientId";
        "client_secret" = $secret;
        "refresh_token" = $refreshToken;
    }

    Write-Host "Refreshing token from $identityServerUrl..."
    $identityServerTokenEndpoint = "$identityServerUrl/identity/connect/token"
    try {
        $tokenString = Invoke-RestMethod -Uri $identityServerTokenEndpoint -Method Post -Body $body
    } catch {
        Write-Host -ForegroundColor Red "Error getting tokens"
        Write-Host -ForegroundColor Red "$($PSItem.Exception)"
        exit 1
    }

    $accessToken = $tokenString.access_token
    $refreshToken = $tokenString.refresh_token
    try {
        & kubectl config set-credentials "$($creds.UserName)" --auth-provider=oidc --auth-provider-arg=idp-issuer-url="$identityServerUrl/identity" --auth-provider-arg=client-id=$clientId --auth-provider-arg=client-secret=$secret --auth-provider-arg=refresh-token=$refreshToken --auth-provider-arg=idp-certificate-authority=$identityCaCrt --auth-provider-arg=id-token=$accessToken
    } catch {
        Write-Host -ForegroundColor Red "Error writing config"
        Write-Host -ForegroundColor Red "$($PSItem.Exception)"
        exit 1
    }

    Write-Host "Update job to include new parameters"
    $oidcJob = Get-ScheduledJob | Where-Object {$_.Name -like "K8s-OIDC-Refresh"}
    Set-ScheduledJob -InputObject $oidcJob -ArgumentList (
        $clientId,
        $secret,
        $refreshToken,
        $identityServerUrl,
        $identityCaCrt,
        $creds
    )
} -ArgumentList (
    $clientId,
    $secret,
    $refreshToken,
    $identityServerUrl,
    $identityCaCrt,
    $creds
)

Now, as long as the computer is on, the scheduled task should keep your Kubernetes configuration file up to date with a fresh token (assuming your user continues to have access). There are obvious downsides with this approach, such as over the weekend if your computer is off, you will need to generate a new token first thing Monday morning. But that is a minor annoyance for good authentication and security.

If you haven't heard, Twitter, Facebook, YouTube, et. al. have recently been “deplatforming” certain people.

Freedom of Speech

Whether or not you were followers of these people, or even listened to them is not relevant. They have all said unpopular things and offended people. But the only speech worth defending is that which is unpopular. Nobody needs to defend popular speech; things that everyone already agrees with. And this is a free speech issue, even though it is not the government participating in censorship. The freedom of expression exists outside of the U.S. Constitution, as a concept and practice.

Yes, these corporations are free to censor who they chose, just as you are able to demand someone you disagree with from leaving your house; this doesn't make it not censorship, though. In the case of these corporations, they control so much of public discourse (at least in the United States), that they must be held to a higher standard (if not by regulation, then by the people directly).

To censor the words of those you disagree with necessarily tends toward an echo chamber. The next leaders of government come from the people in this echo chamber. Therefore it is necessary to hear all the vitriol and hate spewed by the community, and the rational arguments against such speech. To do otherwise is to eventually give the power of government to the corporations that can control the speech of the population.

The people have the power

The Internet is mankind's greatest invention (so far). It has allowed unprecedented communication, innovation, and education. It has prevented wars, and helped cause genocide. It is a powerful tool that can be utilized for good or for evil.

It is a common tactic among governments that wish to go to war to dehumanize the other. We see this with the terms Japs, Huns, Jerry, Kraut, Gooks, Ragheads, etc. We see it with offensive caricatures and cartoons, propaganda depicting the people as dumb or barbaric. Speech is a powerful tool, and utilized by religion, government, or corporations can be used to drive thought in a certain direction. On the other hand, one of the best ways to prevent conflict is to talk.

If it was possible for the people of Nazi Germany, Imperial Japan, Soviet Russia, and the other allied powers to talk freely and instantly, there would be a lot of doubt cast upon this propaganda. For the powers of the state ultimately come from the people. The people build the tanks, pay the tax, and fight the battles. To influence what people can say is to influence what people do.

A technical solution

You probably have not noticed (but props if you have!) that this blog is built on write freely, an open source, federated blog tool. Each blog post is published on @james@jamesekstrom.com, which you can follow on other federated services that speak ActivityPub such as Mastodon (no not the metal band).

ActivityPub is a decentralized social networking protocol. As of writing it is a W3C Recommendation. It allows server-server and client-server communication through “inboxes” and “outboxes”. Anyone can listen on someones outbox to get messages they post, and people can post messages to someone's inbox for them the receive that message. It really is quite simple in concept, but extremely powerful.

To learn more about how ActivityPub works, check out this great blog post by Eugen Rochko on the Mastodon team.

For instance, I said you can receive these blog posts on Mastodon, a micro-blogging tool that is built and maintained by a large group of open source enthusiasts. I got this “exposure” (nobody listens to what I have to say...) for free, simply by using this protocol. Other services such as PeerTube and PixelFed also integrate over ActivityPub, and help form the Fediverse. You can post a video to PeerTube, get a notification on Mastodon, respond to the video post on Mastodon, and the message shows up as a comment on the video. Did you catch that? Let me rephrase. The social part of the social network is not in either PeerTube or Mastodon, but in a level above both: the protocol itself!

Anyone can host an instance of PeerTube, PixelFed, Mastodon, or in my case Write Freely. You could even write your own tool or app that integrates with all of the Fediverse by following a simple protocol. This makes it nearly impossible to censor anything. It gives the power of the Internet and of free communication back to the people, no corporation or government can (easily) censor this (I know there are ways such as DNS blocking or rubber hose interrogation). This is a huge step in the right direction. Away from centralized management and control of speech.

Why it won't catch on

Unfortunately, we must come back to reality. ActivityPub and the greater Fediverse won't catch on for the same it exists in the first place: it is not centrally managed. The network effect actively works against decentralization. Everyone and all your friends are already on <platform>, so that is where you go for content. And to make things worse, these platforms will not implement ActivityPub because it would destroy their user base and revenue stream.

There is also the question of monetization. One of the reasons Twitter is successful is because of advertising money. There is a lot of traffic on Twitter, so advertisers pay good money to get more eyeballs on their ads. If anyone could host their own version of Twitter and receive all the same tweets with no ads, why would anyone visit Twitter? That's not to say it is impossible to make money if you're using ActivityPub, but it dis-incentivizes making money from the social aspect alone.

Hope for the future

I see two possible, although equally unlikely, scenarios in which the Fediverse gains mainstream popularity (say, as popular as Snapchat):

  • A highly used non-social media based app adds social media features using ActivityPub.
  • A highly visible and liked public figure gets deplatformed and brings a large group over to Federated services.

A highly used non-social media based app adds social media features using ActivityPub

The most popular applications I can think of are browsers. Chrome would probably not implement ActivityPub, since it would not interact with YouTube or any other Google social media platform, for the reasons stated above. Firefox could possibly, if Mozilla wanted to incorporate some sort of federated social networking features into the app, but it doesn't have nearly the same market share that Chrome has.

There is also the possibility apps like Dropbox or Spotify or some other widely-used application would want to add social networking capabilities. I don't think this is likely, again because of the network effect. Nobody would use these features because their friends are not using them.

A highly visible and liked public figure gets deplatformed and brings a large group over to Federated services

This user would have to have tens of millions of followers. Someone like Kevin Hart maybe. Almost 100 million instagram followers, about 35 million Twitter followers. Comedian, has been in trouble with the media before. But even if he got banned from Instagram and Twitter, and went to Mastodon, would his followers, ahem, follow him there? Some might, most likely his voice would just no longer be heard on those platforms.

Part of the problem with this outcome is how little publicity the Federverse has (it is pretty new). There are very few tech people with large followings. Elon Musk is probably the most popular tech person in social media (at least in the USA), and I doubt he knows or really cares.

What can we do now?

If, like me, you feel there is a problem with so much centralization on the Internet, then what can we do to help?

Write about it, talk to your friends, talk at your local meetups. Get a conversation going about why it is important, and the dangers of centralized platforms.

Host instances of existing Fediverse projects, and connect to all your followers (this only really works if you already have people following you).

Contribute code. Most Fediverse projects are open source, so that is one avenue. You could also contribute to existing, non-social projects to propose adding some social aspects via ActivtyPub to them. Take a tool that you use frequently that you feel could benefit from publishing notifications to a group of people, for instance. Like RSS. Expand on that a little bit, maybe.

Create new social media applications. Make a federated Snapchat or Meetup application. Make a new social media thing that only exists within the Fediverse.

Summary

Centralized, walled garden variety social networks are dangerous to society. To put so much trust in so few organizations creates fragile systems. Decentralizing speech can help by making it anti-fragile. Free speech must be protected, and we have the tools to help, and with ActivityPub and the Fediverse we can bring the Internet back to the people.


Kubernetes 1.14 brought support for running native Windows containers.

The first step to running a Windows container in your Kubernetes cluster is setting up your node. This Microsoft guide is an excellent resource to get started! Windows Server 2019 brought with it support for container networking (with Flannel), so I'm going to be using that. I'm going to assume a PowerShell interface for all Windows node and pod setup and debugging, though for some debugging it may be easier to connect to the container via a PowerShell session from an RDP instance on the node.

I'm not going to cover setting up Flannel CNI or your Linux master nodes. The Microsoft guide above is a great starting point though.

Throughout this post, I'll try to describe the areas where I had trouble and what I did to resolve it.

Update Windows

There is no easy way to update Windows in a remote PowerShell session (it requires special permissions that are not inherited from the user for some reason), so we set up a scheduled task that runs as System that is set to run once, and then restart the computer.

This could take several minutes, but we want to be on version 1903 or better for container overlay networking support.

$action = New-ScheduledTaskAction -Execute "powershell.exe" -Argument "-Command ipmo PSWindowsUpdate; Get-WUInstall -AcceptAll -Download -Install | Out-File C:\PSWindowsUpdate.log"
$trigger = New-ScheduledTaskTrigger -Once -At (Get-Date)
$principal = New-ScheduledTaskPrincipal "SYSTEM"
$settings = New-ScheduledTaskSettingsSet
$task = New-ScheduledTask -Action $action -Principal $principal -Trigger $trigger -Settings $settings

Register-ScheduledTask t1 -InputObject $task

Start-ScheduledTask t1

Restart-Computer

Install Docker

Of course, we need Docker installed

Install-Module -Name DockerMsftProvider -Repository PSGallery -Force -AcceptLicense -SkipPublisherCheck

Install-Package -Name Docker -ProviderName DockerMsftProvider

Install the Docker images

These Docker images set up the “pause” image that Kubernetes uses to set up the pod's IP address and network namespace. This is a great post on pause containers.

docker pull mcr.microsoft.com/windows/nanoserver:1809

docker tag mcr.microsoft.com/windows/nanoserver:1809 microsoft/nanoserver:latest

docker run microsoft/nanoserver:latest

Prepare for Kubernetes

The guide mentioned in the intro puts everything in a C:\k directory, and I think that is a good convention. We also need to copy over the Kubernetes certificate.

On the node:

mkdir C:\k

On your machine:

$configPath = "C:\path\to\kubernetes\config"
$session = New-PsSession -ComputerName $windowsK8sNode

Copy-Item -path $configPath -Destination C:\k\config -ToSession $session

Downloading and installing the Kubernetes binaries

We need to download kubectl, kube-proxy, and kubelet binaries. You can find the released binaries in the releases for Kubernetes on Github.

We will need 7zip to extract the contents from the .tar.gz file (as of writing, PowerShell has no means to expand this type of archive).

Note: Please verify for yourself all links and binaries before running them!

Invoke-WebRequest https://www.7-zip.org/a/7z1900-x64.msi -outFile 7zip.msi

msiexec.exe /i 7zip.msi

$env:Path += ";C:\Program Files\7-Zip\"

Now download and install the Kubernetes binaries.

Invoke-WebRequest "https://dl.k8s.io/v1.14.0/kubernetes-node-windows-amd64.tar.gz" -outFile kubernetes.tar.gz

7z x .\kubernetes.tar.gz
7z x .\kubernetes.tar

Get-ChildItem kubernetes\node\bin\*.exe |% { Copy-Item $_.FullName C:\k }

rm .\kubernetes.tar 
rm .\kubernetes.tar.gz
rm .\7zip.msi

Verify binaries installed correctly, and the node can access the cluster. Set the PATH environment variable and the KUBECONFIG environment variable so everything knows where to look.

$env:Path += ";C:\k"
[Environment]::SetEnvironmentVariable("Path", $env:Path + ";C:\k", [EnvironmentVariableTarget]::Machine)

$env:KUBECONFIG="C:\k\config"
[Environment]::SetEnvironmentVariable("KUBECONFIG", "C:\k\config", [EnvironmentVariableTarget]::User)

kubectl version

You should not see any errors.

Joining the node to the cluster (using Flannel)

Thankfully Microsoft provides a bootstrapping script that we can take advantage of.

Set-Location C:\k

Invoke-WebRequest https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/start.ps1 -OutFile start.ps1

# Make sure Docker is running
Start-Service docker

.\start.ps1 -ManagementIP <Windows Node IP> -NetworkMode <network mode>  -ClusterCIDR <Cluster CIDR> -ServiceCIDR <Service CIDR> -KubeDnsServiceIP <Kube-dns Service IP> -LogDir <Log directory>

This takes a moment or two. Afterwards, you should be able to run kubectl get nodes and see your Windows node in the cluster.

Tainting the node

To ensure that only Windows containers are scheduled on your Windows node, you can taint your node, and then apply tolerations in your pod specs.

kubectl taint nodes <node name> windows=true:NoSchedule

And then in your pod.yaml or deployment.yaml

tolerations:
  - key: "windows"
    value: "true"
    effect: "NoSchedule"

You could also use nodeSelector, but I found some Linux pods, such as kube-proxy, were sometimes assigned to the Windows node. To prevent this you can taint the node instead of trying to select specific nodes for each pod. Either way would accomplish the same goal.

Other notes and considerations

Running as a service

You can set up kubelet, kube-proxy, and flanneld to run as a Windows service, so a node failure does not require manual intervention. This worked for me:

Set-Location C:\k

Start-BitsTransfer https://nssm.cc/release/nssm-2.24.zip

Expand-Archive .\nssm-2.24.zip nssm-2.24/

Copy-Item nssm-2.24\nssm-2.24\win64\nssm.exe .

Invoke-WebRequest https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/register-svc.ps1 -outFile register-svc.ps1

.\register-svc.ps1 -NetworkMode <Network mode> -ManagementIP <Windows Node IP> -ClusterCIDR <Cluster subnet> -KubeDnsServiceIP <Kube-dns Service IP> -LogDir <Directory to place logs>

Troubleshooting

See the Microsoft guide in the intro for some more troubleshooting, here are some issues I ran into.

My Windows node is NotReady

Make sure that the kubelet process is running on the node: Get-Process -Name kubelet. If you set it up as a Windows service, make sure the service is running: Start-Service kubelet.

My Windows pod cannot communicate with the Internet

Make sure that the pod can hit the kubelet on port 10250. Check Windows firewall on the node and open this port if necessary.

Also make sure that the kube-proxy process is running: Get-Process -Name kube-proxy. If it is not, either run the start script again, or start the service.

Setting up the .NET Framework Container

In the next post, we'll go over how to set up the Windows Dockerfile to run an ASP.NET Framework application in IIS, and some of the issues that came up with that.


Presenting at Twin Ports Web Pros in Duluth!