james at jamesekstrom.com

The Setup

You are in AWS. You cannot let any traffic from the internet, or to the internet. You want a hub-and-spoke architecture with applications connecting through the hub to access API Gateways on the spokes. The VPC Endpoint that facilitates that connection needs to be locked down with a policy in a least privilege manner, allowing only specific roles through the VPC Endpoint.

Desired state

The example endpoint policies only show a user principal, not a role principal, but this is not impossible.

Here is an example policy we want to use:

    "Statement": [
            "Principal": {
                "AWS": [
            "Action": [
            "Effect": "Allow",
            "Resource": [

So only MyRole is allowed across the VPC Endpoint, all other requests will receive a 403 response from the VPC Endpoint.

If you use the AWS CLI to invoke-test-method, passing in the API Gateway id and resource, and a role you have configured, it will work. However this avoids the VPC Endpoint entirely. It just calls the API Gateway in the other account directly, using the apigateway service instead of the execute-api service.

What we want is to make an HTTP request to the private DNS name provided by the VPC and traverse the VPC Endpoint to the private API Gateway.

How to call the API with a role

What we need to do is sign the request using the role that is allowed. AWS API uses a special form of signatures they call Signature Version 4. It's a bit complicated, but nothing we can't whip up in a crude bash script.


The components for this ritual are

  • Access key id
  • Secret access key
  • Token

Once you have all three (perhaps from a aws sts assume-role call), you can stir them together using a series of openssl hashing functions.

After you have generated the signatures, you can finally make your curl request against the API:

curl -H "x-amz-date: ${date}" -H "x-amz-security-token: ${token}" -H "Authorization: ${authorization}" "https://${apiid}.execute-api.${region}.amazonaws.com${apiresource}"

Here is the script, modify it to your use case.



rolejson=`aws sts assume-role --role-arn "<role>" --role-session-name "<sessionname>"`
aws_access_key_id=`echo -en ${rolejson} | grep -Po '"AccessKeyId":.*' | cut -d':' -f2 | sed 's/[^0-9A-Z]*//g'`
aws_secret_access_key=`echo -en ${rolejson} | grep -Po '"SecretAccessKey": "[^\",]*' | cut -d':' -f2 | sed 's/[^0-9A-Za-z/+=]*//g'`
token=`echo -en ${rolejson} | grep -Po '"SessionToken": "[^\",]*' | cut -d':' -f2 | sed 's/[^0-9A-Za-z/+=]*//g'`

# To use ec2 instance profile:
# instance_profile=`curl`
# aws_access_key_id=`curl${instance_profile} | grep AccessKeyid | cut -d':' -f2 | sed 's/[^0-9A-Z]*//g'`
# aws_secret_access_key=`curl${instance_profile} | grep SecretAccessKey | cut -d':' -f2 | sed 's/[^0-9A-Za-z/+=]*//g'`
# token=`curl${instance_profile} | sed -n '/Token/{p;}' | cut -f4 -d'"'`

sdate=$(date + '%Y%m%dT%H%M%SZ')

payload=$(echo -en "" | openssl dgst -sha256 | awk '{print $2}')
canonicalhash=$(echo -en ${canonical} | openssl dgst -sha256 | awk '${print $2}')

dateshort=$(date '+%Y%m%d')

function hmac_sha256 {
    echo -n "$data" | openssl dgst -sha256 -mac HMAC -macopt "$key" | sed 's/^.* //'

dateKey=$(hmac_sha256 key:"AWS4$aws_secret_access_key" $dateshort)
dateRegionKey=$(hmac_sha256 hexkey:$dateKey "us-west-2")
dateRegionServiceKey=$(hmac_sha256 hexkey:$dateRegionKey "execute-api")
signaturekey=$(hmac_sha256 hexkey:$dateRegionServiceKey "aws4_request")

signature=$(echo -en $stringtosign | openssl dgst -sha256 -mac HMAC -macopt hexkey:$signaturekey | awk -F ' ' '{print $2}')

authorization="AWS4-HMAC-SHA256 Credential=${aws_access_key_id}/${dateshort}/us-west-2/execute-api/aws4_request, SignedHeaders=host;x-amz-date;x-amz-security-token, Signature=${signature}"

curl -H "x-amz-date: ${sdate}" -H "x-amz-security-token: ${token}" -H "Authorization: ${authorization}" "https://${apiid}.execute-api.us-west-2.amazonaws.com${apiresource}"

Could there be improvements made to this script? No, it is perfect the way it is.

This took me a few days to figure out, so I hope this can save someone else some time.

The shoulders of giants

I am not a bash wizard, here are some people I copied from:

These tools might help you more than I can.

A donut is just a pastry. Nuance reveals that it's holy.

Strong opinions, weakly held

Nuance is becoming a rare sight on social media. People have strong beliefs, and cutting words with which to defend them. Those insecure with their ideals seem to fight the hardest to defend them. Scared that they may be wrong, they hide from diverse opinions and anything that may sway their mind. No sin is too small, they deal in absolutes; there is no grey area. You're with us or you're against us. No apology is enough to prevent being cancelled.

I believe the earth is flat. What was your first thought after reading that sentence? Challenging your beliefs is uncomfortable. The first reaction is always indignation. I don't really believe in a pancake earth theory, but it is easy to cast the argument of those who do aside, saying “Obviously, Earth is not flat! Didn't you listen in school?” If we take a deeper look and reflect on why someone would think this, it can reinforce our beliefs, but it can also cast light onto shortcomings of our reasoning and opinions we hold. Instead of jumping straight to anger and deriding someone for spreading fake news or conspiracy theories, examine the details. Rebut the false notions with evidence. Find hidden kernels of truth, as uncomfortable as it may feel. These fringe ideas are the very things that the principle of free speech protects, as popular ideas need no protection. Innovation dies in a selfsame sphere of thought.

Ideological proxies

Don't take refuge in the false security of consensus. – Christopher Hitchens

A core tenet of the principle of free speech is the marketplace of ideas. A lively discussion. A give-and-take. Ideas are proposed, refuted, refined, and abandoned in light of new evidence and better thoughts. Abhorrent, baseless claims are quickly shot down by clear reasoning and irrefutable fact.

I still believe this largely to be true, but room for nuance in online discussions is seemingly decreasing. Instead, people communicate with short, pithy comments meant to signal to others which side they're on. It is a lazy form of communication; so void of reason that it can be reduced to 280 or fewer characters. It is easy to join the mob and shout in favor of burning the witch, much harder to ask the mob how they know that she is a witch.

One of the qualities of the human species that stands it apart from other animals is the extent to which we socialize. Humans are inherently social from the moment we are born. A human child depends on its caregivers much longer than any other animal. As part of this dependence is an evolutionary driver for forming tribes and larger social structures. A need to belong is instinctual.

If you want to find happiness, surround yourself with like-minded people. If you have trouble making friends, simply change your ideology. Adherence to normative behavior usually occurs accidentally and gradually. Having your beliefs constantly challenged is a hint that you may be wrong. However, it can also lead people to surround themselves with only similarly wrong-minded people. The real cruelty is it is impossible to determine from within.

If you don't have strong beliefs, find an institution and use them as a proxy. There are many of them, and you likely already belong to one or several. Ideological proxies can include:

  • Religious groups or cults
  • Political parties
  • Activist clubs and groups
  • Fraternities/Sororities

How do you know you're in an ideological proxy organization? Common patterns include: rituals where you “reaffirm your belief” by chanting words, symbols are considered sacred, outsiders are treated with suspicion or outright hate, and to bring up ideas counter to the prevailing thought would be anathema.

Critical thinking is hard. It is easy to say “I'm a democrat” or “I'm protestant” when confronted with questions on your beliefs. It is useful as a starting point for a deeper conversation, but more and more we're not seeing that deeper conversation online. People stop at group-think, the ideas they're supposed to believe in. This leads to people saying “Black Lives Matter” not because they believe in it, but because they have to chant the words of their group. People fear that if they do not say the correct words, they will be labeled racist.

A lack of context

If all you're doing is casting stones, you're probably not going to get very far – Barack Obama

Too often people read only the headlines of articles, ignoring any context. The art of designing a headline to attract attention is refined more each day. With decreased iteration times, A/B testing, and recommendation algorithms, it is easier than ever to design the optimum clickbait. Indeed, we are so inundated with new information that the marketplace for attention spans is in a race to the bottom. With most users viewing content on their mobile device, increasingly their only terminal to the network, there is no incentive to write long articles for an audience of short screens. Content creators instead hone their trade in titillating titles for users to scroll endlessly through.

Of course, the headline is meant to be catchy and exciting, but most users don't actually read the article after the headline catches their eye, sometimes just scrolling through the comments. When you read only the comments you miss the context and get a distorted view of the content. In the worst case subjecting yourself to malicious propaganda.

Context is everything when it comes to meaning. Trying to interpret words without context is like drinking a virgin long island. It's sweet, but you're not going to have a good time. Near the boundaries of context lie the extreme positions. Short slogans that define movements, a motto for a team, a battle cry. Just enough meaning to incite action, but not enough to provide nuance. Rallying behind the rebel yell of a marching protest is a great way get change, but implementation requires a closer look at the issues.

An easy trap to fall into online is responding to the catch phrase instead of the actual position. And the easiest rebuttal to a catch phrase is your own. The thread continues down a spiral of diminishing meaning into more homogenized responses ending in an atomic structure of emogis and retweets, gotchas and ad hominem. Like an overly compressed jpeg, all the detail is lost. Nobody is happy, no minds have been changed. Participants recede back into the comfort of their respective tribe, distilling more extreme views aimed at fomenting rage in the opposition.

Using technology to increase diversity of ideas

So, what can be done to help provide nuanced content? How can one escape an ideological proxy organization and increase critical, independent thought? The answer is obvious, of course: adhere to my ideology.

While the internet makes it easy to find like-minded people, it is similarly easy to find people you disagree with. Exposing yourself to ideas that you find unattractive is just as important as your freedom to express your own ideas. Find these ideas and think about why someone would think such a thing.

Some explicit ways you can use technology to hear new ideas:

  • Subscribe to the RSS feeds of myriad sources (blogs, news, podcasts)
    • RSS is not dead! I use inoreader, but there are many such apps.
  • Write a blog expressing your own nuanced ideas
    • The web may have become too much of a walled garden, but the walls are permeable. The freedom to self publish your own thoughts can not be overstated.
  • Many books can be downloaded and read
    • It may be surprising, but most issues in the world are complex enough to write entire books about.
  • Use sites like https://newscompare.com to identify bias in the news

Sometimes, it is the technology that is hiding nuance. Think carefully of the sites you are using today. Do they increase your exposure to a diversity of ideas? Do they shelter you from adverse opinions? If the latter, consider changing subscriptions, abandoning the platform, or recognize that it is to increase your feeling of belonging and use caution when using it for input to your own opinions.

Keep thinking

Challenge your beliefs often. Think of something you believe but would not feel comfortable saying out loud. Why do you believe that? We have the benefit of viewing those before us through the lens of history. A context that was built from the actions of those living it. Hindsight being what it is, what views do we hold today that will be viewed as heretical or unfashionable in the future? What modern ideas will survive when the pendulum of social doctrine swings back?

As the first generation of internet citizens, it is necessary for a free thinking future that we be good stewards.

A bill in the US congress called the EARN IT Act is making the rounds right now. It attempts to provide a commission with tools to fight against child sexual abuse online. Unfortunately, some of those tools are unconstitutional, including violating the first and fourth amendments. The primary tool it gives authorities is legal force to scan and report on encrypted data, rendering end-to-end encryption de-facto illegal in many cases.

[National Center for Missing and Exploited Children] believes online services should be made to screen their messages for material that NCMEC considers abusive; use screening technology approved by NCMEC and law enforcement; report what they find in the messages to NCMEC; and be held legally responsible for the content of messages sent by others.

This is not the first time, nor will it be the last time a government wishes to make encryption illegal. If it's not the children, then it's the terrorists. The argument usually sounds something like this

We just want to ensure that terrorists do not have a safe space in which to communicate

That was UK Prime Minister David Cameron in 2015.

... but the dangers are real, maintaining law and order in a civilized society is important, protecting our kids is important, so I'd caution against an absolutist view on this.

Obama, SXSW 2016 on end-to-end encryption

If you’re dealing with drug lords, if you’re dealing with terrorists, and if you’re dealing with murderers, I don’t care. We have to find out what’s going on.

Trump, 2020

These types of arguments (appealing to the safety of us or our children) are either disingenuous or show a severe lack of understanding of encryption.

Secret Communications

Most communication people partake in is intended for only the people in the conversation. When you are talking to a friend or significant other, you expect them and only them to listen. You can control who is part of the conversation.

When you later find out the walls in the room were a bit thin, you feel like your privacy was violated. There was a third party in your conversation, unbeknownst to you. Perhaps you talk quieter, or find a new room. Even if you aren't discussing sensitive secrets, whoever is in that other room is not invited to the conversation. Imagine if every time you sent a text message, it was a group chat that included an anonymous third party.

End to end encryption takes the in-person whispering or the finding of a new room into the digital realm. Just as it should not be illegal to whisper, it should not be illegal to encrypt your messages. Nobody should be implicitly invited to a private conversation.

If you make encryption illegal, only criminals can have private conversations

Making something illegal does not stop it from happening. Look at drugs or prostitution. If you make it illegal to have end-to-end chats, then the honest, law abiding citizens will have only public conversations (public in the sense that a third party is listening).

Terrorists and criminals will continue to use encrypted messages. Further, anyone who now wishes to communicate in private is labeled a criminal. This is not only unethical but it does not solve the problem the law is intended to.

There is no way to make backdoors only the good guys can use

One solution that is often touted about is to generate multiple sets of keys. You hold on to one set, and the police hold onto the other. Both sets work to access your secret data. When a warrant is served by a judge, then the police can use their key to search your encrypted communications.

It shouldn't be necessary to state, but the police don't always act in the public's best interest. Local police forces have been guilty of using Stingray devices that mimic cell phone towers for warrantless surveillance. The FBI, NSA, and CIA have long lost any trust towards the American public with warrantless wiretapping, drag nets, and mass surveillance and collection of both metadata and personal records.

Let there be no mistake: if the police hold the keys, there is no private communication. They will be listening, passively collecting data. They will defend it in the name of safety, using the next tragedy as a selling point for further eliminating our fourth amendment protections.

Not to mention the keys to your data are now entrusted to a faceless government agency. If the keys happen to leak, or there is a data breach, then all bets are off. And it would make a mighty juicy target for hackers. A key tenant of cybersecurity is minimizing the attack surface area. Having multiple sets of keys increases the risk of unintended access.

If you have nothing to hide, you have nothing to fear

Why should an honest, law-abiding citizen be worried about law enforcement reading their messages? They have followed all the rules, they have nothing to hide. In fact, it may even work in their favor once the powers-that-be realize what an upstanding citizen they are!

Having nothing to hide is a function of both time and interest. If a federal prosecutor is interested enough in you, they will find something. And when they don't, they can get you for lying to a federal agent or obstruction of justice. Even if they can't find anything on you, just wait for the rules to change. Suddenly, you have something to hide, but it is not hidden. It did not use to be illegal to sell heroin, but try advertising that now. It is not currently illegal to construct a firearm in the US, but perhaps if you wait long enough it will be. You have nothing to hide right now.

Arguing that you don't care about the right to privacy because you have nothing to hide is no different than saying you don't care about free speech because you have nothing to say. – Edward Snowden

For these same reasons, it is common advice to never speak to a police officer any more than necessary. Anything you say can and will be used against you. If your thoughts are in the open, it is much easier to implicate you in a crime whether or not you are guilty.

Freedom of speech

In 2013, Ladar Levison was served a National Security Letter (gag order), and forced to shut down his email service, Lavabit, for refusing to give up the keys to his customer's data. He was tried in a secret court, with dubious representation, and denied his first amendment rights to speak about it.

In 2016, the FBI wanted Apple to create a backdoor in their software to catch the San Bernadino shooter. Apple refused (though the FBI later got access and retrieved worthless data).

Political and societal thoughts are like chaotic winds in a storm. One day they're blowing one direction, the next, there's a 2x4 flying through your window. The principle of free speech only protects the unpopular speech. Laws that restrict government overreach are a very powerful tool for the people, but an equally powerful and complementary tool is encryption.

Encryption cannot be applied in an as-needed basis. It must be ubiquitous, or the very presence of encrypted bits reveals information. Therefore, it is necessary to encrypt as much communication as possible.

Encrypted speech is free speech

Encryption is a mathematical operation on a series of bits, numbers. When we write text on a computer, it is represented as a series of 1s and 0s. This is a fundamental property of information. Some complex math operations are run on this, which is “easy” to run in one direction, but incredibly hard to run in the other, like factoring prime numbers. This allows us to send information without regard to who can read it, as long as your intended recipient has the correct numbers to decipher the true meaning. Also known as asymmetric encryption.

That's pretty neat!

Should it be illegal to communicate certain types of factorization? Or in fact, any type of math? Of course not. Math is a fundamental property of the universe, or at least a clever way of representing the logic of it. This didn't stop the US government from trying to restrict the export of math during the cold war. This was shown to be ridiculous by the printing of t-shirts with the banned math on it.

Chilling effects

An unfortunate side effect of the constant barrage on encryption is that there is a chilling effect of new privacy or security based services being created, specifically in the US. A primary selling point of protonmail, for instance, is that it is hosted in Switzerland, a neutral nation known for strong privacy laws.

A more dangerous chilling effect would be a stigma towards using general purpose secure messaging applications. I have a hard enough time getting people to use Signal, a messenger that uses end-to-end encryption by default. “What do I have to hide?” is commonly asked. This would get even worse if there was a concerted effort to equate using encryption to being a terrorist or pedophile. Thankfully that tactic has not been employed yet, or has not been effective.


There is a threat to private communication. It is cloaked in the good intentions of catching terrorists and pedophiles. Trading a little bit of our liberty for a promise of security. But we must be vigilant against this promise, because it cannot be fulfilled: the premise itself is flawed.

By making it illegal to use end-to-end encryption, we harm only the honest and law-abiding.

By storing copies of our keys, we provide juicy targets and a larger attack surface area.

By creating backdoors, we make front doors for the bad guys.

By limiting encryption we infringe on the fundamental human right to free speech.

Private communication between people cannot be eliminated; mathematics cannot be prohibited, and any attempt to do either is foolish.

Clear Written Communication

With everyone working from home now, written communication has become much more important. Effective and efficient communication is key to working with a team, and it becomes especially important when physical presence is limited or eliminated.

One method I have been using, particularly when writing emails, is to flip everything upside down. Start with the actions that need to be taken, or the results of a test or experiment. Then follow with the current status, next steps, and further details that support the conclusion. This allows readers to parse the content and immediately get the context. The ideas flow from your brain into theirs more quickly. I first was introduced to this concept with an article by Gopal Kapur: “I'm OK; The Bull is Dead”.

A friend of mine recently added another bit to the concept: add deadlines. When describing the action items or results, add a time period in which they should be done. Even if it is a rough estimate, it can put a sense of urgency into the message. Not all messages are urgent, but those that are need to have some action taken. This is like the principles of alerting. An urgent email is like a medium or high priority alert, it needs a human to take action. Providing a timeline for the action to take place can help reinforce the impact.

Strengths of Written Communication

I have always been a big proponent of emails over meetings, and chat rooms in general. There are many benefits to communicating over text to others on the team. It is not a panacea of conversation, of course, there are many reasons to talk in person as well. But by recognizing the strengths of writing, we can leverage them for our advantage.


Talking to a person, using your meat flaps to make sound waves, is an active process by all parties. The listeners need to actively listen and understand what ideas you are presenting, in real time. If there is missing context, the speaker needs to listen to questions. There is a back and forth as ideas and questions, arguments and counters, proposals and rejections are volleyed over the net of cognition in the conversation.

By writing your ideas down, and displaying them to interested parties, you free up the rest of your time to engage in another activity, like implementing other ideas. The interested parties can take their own time to read the text and attempt to understand it. They can set aside time that was otherwise not used. Communicating with text optimizes for total throughput instead of latency or instantaneous bandwidth.


By writing everything down, you have a record of what was said. Documentation. If you use the proper tools and a system of tagging things, this could be made even more useful. But even if you don't, searching through your email or slack channels for what was said can be quite helpful. Forwarding email chains to new members to the team to get them caught up can be quicker than arranging meetings.

A written history can also be useful for liability purposes. You have a paper trail. Something to point to when shit hits the fan and people are playing the blame game. You can prove that the executive signed off on things, or that consensus was reached among the team of engineers, not a hapless individual. Text logs can even be brought up in court for legal ramifications, for better or worse. Depending on the side of the coin you're on, this could be a good or bad thing, granted.


It is much easier to formulate complete, rational, well-thought-out concepts by writing them down instead of speaking them. This is why important speeches are typically read from a teleprompter, or at least well prepared in written form beforehand. A text can be read, edited, and rewritten before sending to its audience. You can discover inconsistencies or contradictions in your reasoning before introducing them to others. You can argue against yourself and write defenses first, saving the time of potential debaters. And the biggest time saver of all: you may find your reasoning fundamentally flawed, and never subject anyone else to it.

A well written argument or proposal can be easier to understand than a simple conversation. There is time to organize the points and optimize the flow of information. Often times in speech, we jump ahead, and have to backtrack to fill in details and context. This can easily confuse people and lead to miscommunication. Spending more time by crafting a well written document can minimize this form of miscommunication.

Chat Rooms

Chat rooms serve a different purpose than email. They are still asynchronous, written communication, but there is an implied shortened time to respond. It's a middle ground between long form writing and speaking directly to someone. Someone in a chat room may be AFK for now, but will respond with a sentence or two when they return 5 minutes later.

Importantly, chat room still have the benefits listed above, if perhaps diminished a little in potency. One advantage that chat rooms often have over email, however, is that there can be hundreds or thousands of listeners (yes, I know mailing lists exist). A small conversation going on in a thread in slack may invoke questions by the silent bystanders, or provide answers to questions they didn't even know to ask.

Most people in chat rooms are lurkers. According to the 1% rule of internet culture, only around 1% of users participate in adding content or engage in the conversation. This doesn't mean that 99% of users are worthless leeches in the chat, even by having only 1% communicating still benefits the whole. The text is written. It can be read, re-read, expanded upon, added to documentation.


One of the big advantages of chat rooms is bots. A chat bot can be useful, like alerting the room when a deployment was kicked off or a build failed. Or it can be light hearted, like automatically finding gifs on the internet related to a phrase. Both of these types of bots are popular for most slack teams.

A bot can also be interactive. Perhaps instead of just receiving information that a build was started, you want to actually start the build from the chat room. This is also quite useful for some teams. This and a lot of other techniques are covered in ChatOps, which can allow teams to collaborate in a conversation to orchestrate workflows.

Rhyme Bot

Less useful, maybe, are bots that respond in rhyme. Imagine a fun, harmless chat bot that responds with a line from song lyrics based on the last word said (if prompted). For example:

[Bob]: rhyme these chips have a lot of flavor

[RhymeBot]: A rhythm recipe that you'll savor

Pretty simple, can be fun trying to figure out which track the rhymed lyrics come from.

I don't have to imagine. I wrote this bot a few years ago, naively. Didn't see any issues with it. It was fun for a little bit. But it turns out, that if you feed the bot a lot of rap music lyrics, and then rhyme with 'bigger' or 'trigger', for example, the results might not be exactly work-friendly. They're going to be exactly what you think they are. You didn't even have to prompt the bot to rhyme with '-igger', there are so many songs out there with NSFW lyrics (depending on where you work). It may be fine to listen to them, but to have them written out in your chat room doesn't look great.


I took the bot down after a couple days (and a talking with my boss). Chat etiquette is a little different than speech. I think it is easier to get away with some stuff when talking because it is ephemeral. Sometimes you say something and it seems to “hang” in the air when nobody really knows how to respond. With text that is amplified. The words are right there, and everyone keeps reading them over again.

Some of my personal thoughts on general etiquette, in a professional setting:

  • If you don't have anything to add to the conversation, don't write anything.
  • If you're in a chat with a bunch of people that you don't know well personally, try not to offend.
  • Keep more involved conversations in separate channels or threads
  • If you find something funny on the internet, and you really have to share, only share it in the correct chat.
  • Encouraging writing fun bots is good, but maybe vet them before using them with your entire team


Yes, I didn't practice the upside-down email logic to this blog post. That is intentional. I am sharing a story, a train of thought. I'm not trying to convince anyone of any particular set of actions. It is prudent to gauge your audience when crafting any form of communication.

If there is one thing to be gained by more remote work, it is better reasoned, clearer, and more concise written thoughts. This will serve us now and in the future when we can all return to the office. The practices engaged in when writing to each other transfers easily to documentation, speech writing, and the general public conversation over the internet and social media.

Learning effective communication on a remote team is crucial. It is imperative that each member on the team talk, and the first tool for that job is text chat and email. They each are often the best tools, especially when used properly.

And most importantly: bots might not see color, but can lead to uncomfortable conversations with HR.

Serverless is an architecture pattern that allows the user to not worry about servers. Of course servers still exist in a data center somewhere, but they do not concern the customer. These are many of the benefits to using this pattern.


“All problems in computer science can be solved by another level of indirection... Except for the problem of too many layers of indirection.” — David Wheeler

Serverless architecture abstracts the hardware from the software. In the same way that an operating system can provide a framework and environment without the user having to know where to put things in memory, the cloud allows you to write software without having to worry about how it is actually running.

Abstractions are useful because they allow us to think at a higher level. We can think in the solution space, rather than in unnecessary details like config file management and patching regimens. Many people have wasted many hours tracing down that config setting in their load balancer, and dealing with rolling out patches gradually so as to not interrupt service.

These things, and many more like it, must still happen, but they are done by experts, or at least people dedicated to doing that as their job. In Domain Driven Design we would classify this as a generic sub-domain; something that is required, but not core to the business goals. By abstracting the generic management stuff away, you are left with only the problem core to the business, which allows faster iteration on the solution that will generate income.

There are a lot of benefits to abstracting the server infrastructure out of your stack. If you are able to leverage the cloud completely to reach your business goal or solve your problem, you can significantly reduce cognitive overhead and maintenance, and increase development velocity.

Cognitive Overhead

When running a traditional infrastructure, there is a significant need to have a decently accurate mental model. Most production infrastructures are complex enough that it is nearly impossible to know every bit of it in sufficient detail. Some of this doesn't go away with serverless. Here are some of the benefits, in my mind:

In a traditional infrastructure you need to be able to predict and provision resources ahead of time so you don't go down. Serverless infrastructure is immutable by default and since you don't provision servers, and you don't have configuration files to manage, there are fewer moving parts. A traditional infrastructure could be set up in an immutable fashion, but you must go out of your way to set that up and to maintain it. Instead of thinking how many servers you need, you think in terms of what resources and services help you solve your problem.

A large part of the cognitive overhead of running a traditional infrastructure is maintenance. You need to be able to keep your servers up, running, and patched.


Maintenance is a big part of any robust infrastructure. With serverless you can minimize it. There are no servers to keep patched and updated, there are no application configuration files you need to store in puppet or chef.

There is still maintenance you need to perform for serverless. You may need to optimize functions and queries, manage log tracing and analytics, and prune unused resources occasionally.

Even with serverless, there is still the issue of the cloud provider messing up. Google and Amazon engineers are not infallible, and have often made configuration errors that resulted in downtime. Examples: Google storage downtime AWS S3 downtime

Most cloud resources have a lot of 9's of reliability, however. AWS S3 for instance is has 99.99% availability and 99.999999999% durability over a year. This means that there should be no more than 52 minutes of downtime, and if you store 10,000,000 objects you could expect to lose an object every 10,000 years. The experts at the cloud provider work hard to keep the backend up, so you don't have to. Your company doesn't have to have as much staff on hand to keep things running.

Scale to Zero

Scale to Zero is the concept of having no provisioned resources when you have no traffic. This has a large benefit of saving a ton of money, and not being wasteful. However, a downside is that there can be a lag between when a request comes in and resources are available to handle the request.

In a traditional infrastructure you would have to provision enough resources to handle the peak load. This means that (taking the 80-20 rule for sake of argument), you are not using your resources to their capacity 80% of the time. Yet you must pay to run these servers. Electricity must be used, coal burned, turbines spun, and the only output is unnecessary heat.

In aggregate at cloud scale, serverless resources can be provisioned dynamically and much more dense to use nearly 100% of the available power. Of course in reality the datacenter must keep a lot of hardware in standby ready to take up the load, however, this is minimized in the case of serverless.

Here is a very detailed graph of this concept, as an example. pretty graph showing provisioned resources as a constant line, while serverless resources follow the curve of user traffic

I don't think everyone should rush to implement serverless architectures to try to save the planet – there are better reasons to go serverless – but the optimization in cost should be considered. Because you only pay for what you use, your cost structure can be related more directly to your usage patterns. If you are just starting out and have few users, your costs will be low. Your costs scale with you, and can be factored in to the pricing model, ideally.

Serverless scales to your credit card, and this allows you to focus on what really matters to the business.

Don't reinvent the wheel, focus on business value

There is a lot of technology involved in large enterprise infrastructure projects. A lot of networking, load balancing, firewalls, general logic, and things otherwise unrelated to generating income for your business, or unrelated to the problem you are solving.

So many of these parts of the infrastructure are not necessary to provision and manage with a serverless architecture. They have been done for you, there is no need to waste time and reinvent the wheel. This immediately gives businesses that adopt serverless from the start a leg-up. The engineers can focus on core value.

With a good software architecture and corresponding business architecture (Conway's Law), your development velocity should be much higher. There are fewer barriers between writing the software and delivering it to your customers.

Speaking of Conway's Law, a serverless pattern leans heavily into microservices by default. To implement a well architected solution requires a business organization that is complementary. This can have many benefits such as faster iterations with smaller teams, and fewer bugs since changesets are smaller.

Future Impact

There is no current standard for serverless computing. The International Data Center Authority mentions it in their AE360 paper (the compute layer), but this is not a robust standard.Because there is no standard, you have to decide between the three major cloud providers, or roll your own. There is the Open FaaS project that can help with functions, and you could pair this together other SaaS products, but you would need to write a lot of integration code.

There is a real concern of vendor lock-in, which really limits the ability to migrate, and introduces a bit of unpredictability. If serverless architecture is going to be the future, I expect more standardization around common patterns will be necessary. Another huge benefit to standardization is consolidation of tooling. Terraform providers, and each cloud's specific tools are great, but they lack features and are an obstacle to deploying infrastructure as code. This is also how cloud providers could benefit from standardization.

Things I hope to see standardized:

  • Compute
  • API
  • Storage
  • Caching
  • CDN
  • Messaging/Eventing
  • Monitoring
  • Analytics

Outside of standardization, I see more layers of abstraction being built up. For instance, in AWS, why should the user decide between RDS Aurora, DynamoDB, or Redshift? If there was a standard for serverless storage, a user could identify their access patterns and requirements, and have the proper infrastructure provided automatically.

Another method of abstraction would be to remove the notion of regions. Infrastructure could be moved around seamlessly to serve patterns of traffic automatically. Automatically provisioning instances of compute and replicating data to storage that is closer to the user.

Regardless of standardization or further abstractions, serverless is set to have a huge impact on computing. The industry is still growing rapdily, and more and more companies are moving to the cloud and going serverless.

Arguments Against

There are a lot of reasons why you should not go serverless. Vendor lock in, cost, knowledge, Capex vs Opex, and it may just not provide the functionality that is required for your problem or business.

These are all valid concerns and need to be considered when planning your next architecture or migration.

Vendor Lock In

This is the fear of putting all your eggs in one basket. Now that all of your infrastructure is in AWS, what happens when AWS raises their prices or removes a product that you are depending on? How quickly can you migrate away from your 14 TB RDS instance?

This fear is alleviated somewhat with more cloud providers. Competition provides a bit of incentive for prices to remain low and predictable, because it would be easy to undercut for similar or equivalent products.

With IaC, it becomes easier to replicate infrastructure in a competing cloud environment while still running in the existing one. Migrating your data is a trickier issue, but with careful planning and services like AWS Snowball, it is definitely possible.

There are also hybrid solutions. Perhaps you balance your infrastructure across multiple cloud environments. This is tricky to set up, and maintenance heavy, but if vendor lock in is a major concern, this is one alternative.


Cost can be make it or break it for a business just starting out, or one with tight margins. If you have a new business with unknown usage patterns, however, serverless can be a perfect fit. Since you can scale to zero, you can save a lot of money when your customers aren't using your platform.

If you are a business with tight margins, you probably have well-known and predictable access patterns. This is one case where serverless may not be the solution. If you can provision ahead of time the exact amount of resources required, then you can save a lot of money.

However, this allows no room for growth, at least not rapidly. You could find yourself in a position where you just can't handle the load the users are putting on your system. In that case, automatically scaling with cloud infrastructure could keep your business alive and making money.

A hybrid solution in that case may be the perfect fit. You have provisioned resources for the base load, but can burst into the cloud when necessary to handle surges of traffic. This requires some careful planning, but is definitely possible, though this strays away from serverless a bit.


Knowledge can easily be a barrier to entry to new technology. New things take time and persistence to learn. You may be hesitant to jump into serverless because there is just too much going on, and things are working just fine right now, thank you very much.

Fortunately, there are a lot of resources for learning this technology. From acloud.guru to linuxacademy, and all of the certificate training courses, you have not only the teaching material, but also the accreditation to show that you learned it.

One of the huge benefits to the computer software industry is that anyone can learn it if they put the time and effort it. It is a great equalizer. But it does require the effort.

Capex vs Opex

Your company may be unwilling to migrate to serverless, or the cloud in general, because it is not a capital expense. The company does not own anything for equity purposes, they are merely renting hardware. I am not a financial advisor or an accountant, but this seems like a silly argument. Any money saved by clever accounting for tax purposes could be overshadowed by the amount of money saved by having a more efficient infrastructure. Your company makes its money from software, and that software needs to run on something.

You can pay people money to maintain and build that infrastructure, or you could pay a third party to do it for you. There is a good chance that your staff of infrastructure, network, and software engineers to build and maintain that will be more expensive.

Missing functionality

If you have a niche business or scientific experiment, you may not find the features you require in the cloud. There may be special requirements for privacy, security, or strange computations that just aren't there. In some cases this is absolutely true.

For instance, I don't see a quantum computer class of EC2 instances coming to AWS any time soon. In a lot of other cases, however, reframing the problem to use the provided tools is possible, and may even lead to new, more efficient solutions.

Perhaps you do not need to calculate everything at once in a large simulation, and an eventually consistent event-driven architecture would get the job done as well. This is more of a case-by-case thing, but it is worth considering alternative approaches because they could turn out to be more cost effective and efficient.


Serverless is a great form of computing, and it is here to stay. The industry is still growing rapidly, and for good reason. Companies can save a lot of money and time by using only the resources they need, and cutting out self-hosted infrastructure solutions.

Money can be saved not only in total run time costs, but also in manpower. Staffing requirements can be lessened, or people can be moved onto solving the business goals that actually generate income.

Serverless is a great abstraction on infrastructure in general. It provides the tools to think in the solution space, instead of all the steps required to set it up.

An integral part of running a Kubernetes cluster in production is security, including how human users authenticate with the API server (the public interface of the cluster). A properly configured cluster should allow only those with proper credentials access. And those with access should only have the ability to interact with the objects necessary to complete their jobs. This is the principle of least privilege. I may go into RBAC in Kubernetes later, if I feel like I have anything interesting to say on the topic. For now, let's see how authentication can work with OIDC.

The Kubernetes documentation has a very good article on Authenticating which I suggest you read for more in-depth knowledge on this topic. I am focusing on the OpenID Connect Tokens section. I am going to assume some basic understanding of OAuth2 and OIDC flows. If you are not familiar, here is a good illustrated guide I saw recently come up on Hacker News.

Configuring the Kubernetes API server for OIDC

The first step in this process is configuring the API server for OpenID Connect. There are several arguments that need to be passed into the server on startup to get this all wired in. You can see the full list here, but I am going to mention one:


This argument defaults to the sub value, which may work for you. I changed this to the email field for it to be a little more user friendly when assigning UserRoleBindings later on (it makes it clear who is gaining access to what, especially if you are using infrastructure as code practices).

Once you have configured the client id, URL, and CA for your OIDC provider (idp), you can get a token and attempt to connect.

Getting the token from your identity provider

To retrieve the token from your identity provider, you will need a few things:

  • client id
  • client secret
  • grant type
  • scopes requested
  • username
  • password

Let's do this with some PowerShell

$creds = Get-Credential -Message "Provide your $identityServerUrl credentials."

This will generate a prompt for the user to enter in their username and password, and stores the password in a System.Security.SecureString object. This ensures that the data is removed from memory properly when no longer needed (of course a clever user who has access to the system would probably be able to get the data if they really wanted, but it indicates that this is sensitive information).

Now let's create the body for the request

$body = @{
  "scope" = "kubernetes email_only offline_access";
  "grant_type" = "password";
  "client_id" = "$clientId";
  "client_secret" = $secret;
  "username" = $creds.UserName;
  "password" = $($creds.GetNetworkCredential().Password);

You will have to fill in the details with your own idp. You can also use different scopes if you wish. The kubernetes scope was one I added to handle specific information for Kubernetes based authentication.

$identityServerTokenEndpoint = "$identityServerUrl/identity/connect/token"
$tokenString = Invoke-RestMethod -Uri $identityServerTokenEndpoint -Method Post -Body $body

Now we can call the token endpoint to get our token from the identity provider. The response should have in it an access token and a refresh token (it may have an id_token instead, depending on identity provider). We can use the access token as the identity token in our Kubernetes configuration file. We can set our .kube config using kubectl, and then use the new context to authenticate with our cluster:

$accessToken = $tokenString.access_token
$refreshToken = $tokenString.refresh_token

$clusterName = "kubernetes-idp"
$contextName = "$clusterName"

& kubectl config set-cluster $clusterName --server=$k8sServerUrl --certificate-authority=$k8sCaCrt

& kubectl config set-context $contextName --cluster=$clusterName --user="$($creds.UserName)"

& kubectl config set-credentials "$($creds.UserName)" --auth-provider=oidc --auth-provider-arg=idp-issuer-url="$identityServerUrl/identity" --auth-provider-arg=client-id=$clientId --auth-provider-arg=client-secret=$secret --auth-provider-arg=refresh-token=$refreshToken --auth-provider-arg=idp-certificate-authority=$identityCaCrt --auth-provider-arg=id-token=$accessToken

& kubectl config use-context $contextName

Where $identityCaCrt is the identity provider's CA for its public domain certificate. The other variables should be self-explanatory. Just fill in the details of your cluster.

The token returned by the identity provider is a JWT. One downside with JWTs is that there is no mechanism to invalidate them directly (because the client controls the certificate, it is an asynchronous process by design). We can alleviate concerns by setting a short TTL on these tokens, but this leads to another issue: it can be annoying to keep getting these tokens. This is what the refresh token is for.

Setting the refresh token should help with generating valid credentials while this token is being used. However, if you go an extended period without using your token, it will be invalid the next time you try to use it. One way to help with this issue is a scheduled task or cronjob to continually generate these credentials and update your Kubernetes config file.

Refresh the token

To refresh the token, we make a refresh token request to the identity provider, using the refresh token we got initially.

$body = @{
  "grant_type" = "refresh_token";
  "client_id" = "$clientId";
  "client_secret" = $secret;
  "refresh_token" = $refreshToken;

$identityServerTokenEndpoint = "$identityServerUrl/identity/connect/token"
$tokenString = Invoke-RestMethod -Uri $identityServerTokenEndpoint -Method Post -Body $body

As an example to register a scheduled task on Windows that runs every 50 minutes (assuming the TTL on the token is 60 minutes):

$trigger = New-JobTrigger -Once -At ((Get-Date).AddMinutes(50)) -RepetitionInterval (New-TimeSpan -Minute 50) -RepeatIndefinitely
Register-ScheduledJob -Name "K8s-OIDC-Refresh" -Trigger $trigger -ScriptBlock {

    $body = @{
        "grant_type" = "refresh_token";
        "client_id" = "$clientId";
        "client_secret" = $secret;
        "refresh_token" = $refreshToken;

    Write-Host "Refreshing token from $identityServerUrl..."
    $identityServerTokenEndpoint = "$identityServerUrl/identity/connect/token"
    try {
        $tokenString = Invoke-RestMethod -Uri $identityServerTokenEndpoint -Method Post -Body $body
    } catch {
        Write-Host -ForegroundColor Red "Error getting tokens"
        Write-Host -ForegroundColor Red "$($PSItem.Exception)"
        exit 1

    $accessToken = $tokenString.access_token
    $refreshToken = $tokenString.refresh_token
    try {
        & kubectl config set-credentials "$($creds.UserName)" --auth-provider=oidc --auth-provider-arg=idp-issuer-url="$identityServerUrl/identity" --auth-provider-arg=client-id=$clientId --auth-provider-arg=client-secret=$secret --auth-provider-arg=refresh-token=$refreshToken --auth-provider-arg=idp-certificate-authority=$identityCaCrt --auth-provider-arg=id-token=$accessToken
    } catch {
        Write-Host -ForegroundColor Red "Error writing config"
        Write-Host -ForegroundColor Red "$($PSItem.Exception)"
        exit 1

    Write-Host "Update job to include new parameters"
    $oidcJob = Get-ScheduledJob | Where-Object {$_.Name -like "K8s-OIDC-Refresh"}
    Set-ScheduledJob -InputObject $oidcJob -ArgumentList (
} -ArgumentList (

Now, as long as the computer is on, the scheduled task should keep your Kubernetes configuration file up to date with a fresh token (assuming your user continues to have access). There are obvious downsides with this approach, such as over the weekend if your computer is off, you will need to generate a new token first thing Monday morning. But that is a minor annoyance for good authentication and security.

If you haven't heard, Twitter, Facebook, YouTube, et. al. have recently been “deplatforming” certain people.

Freedom of Speech

Whether or not you were followers of these people, or even listened to them is not relevant. They have all said unpopular things and offended people. But the only speech worth defending is that which is unpopular. Nobody needs to defend popular speech; things that everyone already agrees with. And this is a free speech issue, even though it is not the government participating in censorship. The freedom of expression exists outside of the U.S. Constitution, as a concept and practice.

Yes, these corporations are free to censor who they chose, just as you are able to demand someone you disagree with from leaving your house; this doesn't make it not censorship, though. In the case of these corporations, they control so much of public discourse (at least in the United States), that they must be held to a higher standard (if not by regulation, then by the people directly).

To censor the words of those you disagree with necessarily tends toward an echo chamber. The next leaders of government come from the people in this echo chamber. Therefore it is necessary to hear all the vitriol and hate spewed by the community, and the rational arguments against such speech. To do otherwise is to eventually give the power of government to the corporations that can control the speech of the population.

The people have the power

The Internet is mankind's greatest invention (so far). It has allowed unprecedented communication, innovation, and education. It has prevented wars, and helped cause genocide. It is a powerful tool that can be utilized for good or for evil.

It is a common tactic among governments that wish to go to war to dehumanize the other. We see this with the terms Japs, Huns, Jerry, Kraut, Gooks, Ragheads, etc. We see it with offensive caricatures and cartoons, propaganda depicting the people as dumb or barbaric. Speech is a powerful tool, and utilized by religion, government, or corporations can be used to drive thought in a certain direction. On the other hand, one of the best ways to prevent conflict is to talk.

If it was possible for the people of Nazi Germany, Imperial Japan, Soviet Russia, and the other allied powers to talk freely and instantly, there would be a lot of doubt cast upon this propaganda. For the powers of the state ultimately come from the people. The people build the tanks, pay the tax, and fight the battles. To influence what people can say is to influence what people do.

A technical solution

You probably have not noticed (but props if you have!) that this blog is built on write freely, an open source, federated blog tool. Each blog post is published on @james@jamesekstrom.com, which you can follow on other federated services that speak ActivityPub such as Mastodon (no not the metal band).

ActivityPub is a decentralized social networking protocol. As of writing it is a W3C Recommendation. It allows server-server and client-server communication through “inboxes” and “outboxes”. Anyone can listen on someones outbox to get messages they post, and people can post messages to someone's inbox for them the receive that message. It really is quite simple in concept, but extremely powerful.

To learn more about how ActivityPub works, check out this great blog post by Eugen Rochko on the Mastodon team.

For instance, I said you can receive these blog posts on Mastodon, a micro-blogging tool that is built and maintained by a large group of open source enthusiasts. I got this “exposure” (nobody listens to what I have to say...) for free, simply by using this protocol. Other services such as PeerTube and PixelFed also integrate over ActivityPub, and help form the Fediverse. You can post a video to PeerTube, get a notification on Mastodon, respond to the video post on Mastodon, and the message shows up as a comment on the video. Did you catch that? Let me rephrase. The social part of the social network is not in either PeerTube or Mastodon, but in a level above both: the protocol itself!

Anyone can host an instance of PeerTube, PixelFed, Mastodon, or in my case Write Freely. You could even write your own tool or app that integrates with all of the Fediverse by following a simple protocol. This makes it nearly impossible to censor anything. It gives the power of the Internet and of free communication back to the people, no corporation or government can (easily) censor this (I know there are ways such as DNS blocking or rubber hose interrogation). This is a huge step in the right direction. Away from centralized management and control of speech.

Why it won't catch on

Unfortunately, we must come back to reality. ActivityPub and the greater Fediverse won't catch on for the same it exists in the first place: it is not centrally managed. The network effect actively works against decentralization. Everyone and all your friends are already on <platform>, so that is where you go for content. And to make things worse, these platforms will not implement ActivityPub because it would destroy their user base and revenue stream.

There is also the question of monetization. One of the reasons Twitter is successful is because of advertising money. There is a lot of traffic on Twitter, so advertisers pay good money to get more eyeballs on their ads. If anyone could host their own version of Twitter and receive all the same tweets with no ads, why would anyone visit Twitter? That's not to say it is impossible to make money if you're using ActivityPub, but it dis-incentivizes making money from the social aspect alone.

Hope for the future

I see two possible, although equally unlikely, scenarios in which the Fediverse gains mainstream popularity (say, as popular as Snapchat):

  • A highly used non-social media based app adds social media features using ActivityPub.
  • A highly visible and liked public figure gets deplatformed and brings a large group over to Federated services.

A highly used non-social media based app adds social media features using ActivityPub

The most popular applications I can think of are browsers. Chrome would probably not implement ActivityPub, since it would not interact with YouTube or any other Google social media platform, for the reasons stated above. Firefox could possibly, if Mozilla wanted to incorporate some sort of federated social networking features into the app, but it doesn't have nearly the same market share that Chrome has.

There is also the possibility apps like Dropbox or Spotify or some other widely-used application would want to add social networking capabilities. I don't think this is likely, again because of the network effect. Nobody would use these features because their friends are not using them.

A highly visible and liked public figure gets deplatformed and brings a large group over to Federated services

This user would have to have tens of millions of followers. Someone like Kevin Hart maybe. Almost 100 million instagram followers, about 35 million Twitter followers. Comedian, has been in trouble with the media before. But even if he got banned from Instagram and Twitter, and went to Mastodon, would his followers, ahem, follow him there? Some might, most likely his voice would just no longer be heard on those platforms.

Part of the problem with this outcome is how little publicity the Federverse has (it is pretty new). There are very few tech people with large followings. Elon Musk is probably the most popular tech person in social media (at least in the USA), and I doubt he knows or really cares.

What can we do now?

If, like me, you feel there is a problem with so much centralization on the Internet, then what can we do to help?

Write about it, talk to your friends, talk at your local meetups. Get a conversation going about why it is important, and the dangers of centralized platforms.

Host instances of existing Fediverse projects, and connect to all your followers (this only really works if you already have people following you).

Contribute code. Most Fediverse projects are open source, so that is one avenue. You could also contribute to existing, non-social projects to propose adding some social aspects via ActivtyPub to them. Take a tool that you use frequently that you feel could benefit from publishing notifications to a group of people, for instance. Like RSS. Expand on that a little bit, maybe.

Create new social media applications. Make a federated Snapchat or Meetup application. Make a new social media thing that only exists within the Fediverse.


Centralized, walled garden variety social networks are dangerous to society. To put so much trust in so few organizations creates fragile systems. Decentralizing speech can help by making it anti-fragile. Free speech must be protected, and we have the tools to help, and with ActivityPub and the Fediverse we can bring the Internet back to the people.

Kubernetes 1.14 brought support for running native Windows containers.

The first step to running a Windows container in your Kubernetes cluster is setting up your node. This Microsoft guide is an excellent resource to get started! Windows Server 2019 brought with it support for container networking (with Flannel), so I'm going to be using that. I'm going to assume a PowerShell interface for all Windows node and pod setup and debugging, though for some debugging it may be easier to connect to the container via a PowerShell session from an RDP instance on the node.

I'm not going to cover setting up Flannel CNI or your Linux master nodes. The Microsoft guide above is a great starting point though.

Throughout this post, I'll try to describe the areas where I had trouble and what I did to resolve it.

Update Windows

There is no easy way to update Windows in a remote PowerShell session (it requires special permissions that are not inherited from the user for some reason), so we set up a scheduled task that runs as System that is set to run once, and then restart the computer.

This could take several minutes, but we want to be on version 1903 or better for container overlay networking support.

$action = New-ScheduledTaskAction -Execute "powershell.exe" -Argument "-Command ipmo PSWindowsUpdate; Get-WUInstall -AcceptAll -Download -Install | Out-File C:\PSWindowsUpdate.log"
$trigger = New-ScheduledTaskTrigger -Once -At (Get-Date)
$principal = New-ScheduledTaskPrincipal "SYSTEM"
$settings = New-ScheduledTaskSettingsSet
$task = New-ScheduledTask -Action $action -Principal $principal -Trigger $trigger -Settings $settings

Register-ScheduledTask t1 -InputObject $task

Start-ScheduledTask t1


Install Docker

Of course, we need Docker installed

Install-Module -Name DockerMsftProvider -Repository PSGallery -Force -AcceptLicense -SkipPublisherCheck

Install-Package -Name Docker -ProviderName DockerMsftProvider

Install the Docker images

These Docker images set up the “pause” image that Kubernetes uses to set up the pod's IP address and network namespace. This is a great post on pause containers.

docker pull mcr.microsoft.com/windows/nanoserver:1809

docker tag mcr.microsoft.com/windows/nanoserver:1809 microsoft/nanoserver:latest

docker run microsoft/nanoserver:latest

Prepare for Kubernetes

The guide mentioned in the intro puts everything in a C:\k directory, and I think that is a good convention. We also need to copy over the Kubernetes certificate.

On the node:

mkdir C:\k

On your machine:

$configPath = "C:\path\to\kubernetes\config"
$session = New-PsSession -ComputerName $windowsK8sNode

Copy-Item -path $configPath -Destination C:\k\config -ToSession $session

Downloading and installing the Kubernetes binaries

We need to download kubectl, kube-proxy, and kubelet binaries. You can find the released binaries in the releases for Kubernetes on Github.

We will need 7zip to extract the contents from the .tar.gz file (as of writing, PowerShell has no means to expand this type of archive).

Note: Please verify for yourself all links and binaries before running them!

Invoke-WebRequest https://www.7-zip.org/a/7z1900-x64.msi -outFile 7zip.msi

msiexec.exe /i 7zip.msi

$env:Path += ";C:\Program Files\7-Zip\"

Now download and install the Kubernetes binaries.

Invoke-WebRequest "https://dl.k8s.io/v1.14.0/kubernetes-node-windows-amd64.tar.gz" -outFile kubernetes.tar.gz

7z x .\kubernetes.tar.gz
7z x .\kubernetes.tar

Get-ChildItem kubernetes\node\bin\*.exe |% { Copy-Item $_.FullName C:\k }

rm .\kubernetes.tar 
rm .\kubernetes.tar.gz
rm .\7zip.msi

Verify binaries installed correctly, and the node can access the cluster. Set the PATH environment variable and the KUBECONFIG environment variable so everything knows where to look.

$env:Path += ";C:\k"
[Environment]::SetEnvironmentVariable("Path", $env:Path + ";C:\k", [EnvironmentVariableTarget]::Machine)

[Environment]::SetEnvironmentVariable("KUBECONFIG", "C:\k\config", [EnvironmentVariableTarget]::User)

kubectl version

You should not see any errors.

Joining the node to the cluster (using Flannel)

Thankfully Microsoft provides a bootstrapping script that we can take advantage of.

Set-Location C:\k

Invoke-WebRequest https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/start.ps1 -OutFile start.ps1

# Make sure Docker is running
Start-Service docker

.\start.ps1 -ManagementIP <Windows Node IP> -NetworkMode <network mode>  -ClusterCIDR <Cluster CIDR> -ServiceCIDR <Service CIDR> -KubeDnsServiceIP <Kube-dns Service IP> -LogDir <Log directory>

This takes a moment or two. Afterwards, you should be able to run kubectl get nodes and see your Windows node in the cluster.

Tainting the node

To ensure that only Windows containers are scheduled on your Windows node, you can taint your node, and then apply tolerations in your pod specs.

kubectl taint nodes <node name> windows=true:NoSchedule

And then in your pod.yaml or deployment.yaml

  - key: "windows"
    value: "true"
    effect: "NoSchedule"

You could also use nodeSelector, but I found some Linux pods, such as kube-proxy, were sometimes assigned to the Windows node. To prevent this you can taint the node instead of trying to select specific nodes for each pod. Either way would accomplish the same goal.

Other notes and considerations

Running as a service

You can set up kubelet, kube-proxy, and flanneld to run as a Windows service, so a node failure does not require manual intervention. This worked for me:

Set-Location C:\k

Start-BitsTransfer https://nssm.cc/release/nssm-2.24.zip

Expand-Archive .\nssm-2.24.zip nssm-2.24/

Copy-Item nssm-2.24\nssm-2.24\win64\nssm.exe .

Invoke-WebRequest https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/register-svc.ps1 -outFile register-svc.ps1

.\register-svc.ps1 -NetworkMode <Network mode> -ManagementIP <Windows Node IP> -ClusterCIDR <Cluster subnet> -KubeDnsServiceIP <Kube-dns Service IP> -LogDir <Directory to place logs>


See the Microsoft guide in the intro for some more troubleshooting, here are some issues I ran into.

My Windows node is NotReady

Make sure that the kubelet process is running on the node: Get-Process -Name kubelet. If you set it up as a Windows service, make sure the service is running: Start-Service kubelet.

My Windows pod cannot communicate with the Internet

Make sure that the pod can hit the kubelet on port 10250. Check Windows firewall on the node and open this port if necessary.

Also make sure that the kube-proxy process is running: Get-Process -Name kube-proxy. If it is not, either run the start script again, or start the service.

Setting up the .NET Framework Container

In the next post, we'll go over how to set up the Windows Dockerfile to run an ASP.NET Framework application in IIS, and some of the issues that came up with that.

Presenting at Twin Ports Web Pros in Duluth!