package com.snuxoll.blog;

On “DevOps”

DevOps is a very “hip” word right now, but I’m beginning to feel like it doesn’t mean what I believe it should. I keep seeing “DevOps” jobs, or pushes to make a Systems team learn how to write puppet modules to “automate” their infrastructure; often I see people try to simplify DevOps down to “infrastructure as code”.

I also see this notion that Systems Administrators need to learn to “program”, or they will be out of a job. I think that’s absolutely ludicrous, as a software developer who also dons the sysadmin hat occasionally I think we do not give enough credit to the amount of work Systems teams have outside of installing software and deploying new servers.

Wikipedia has a great description of DevOps

DevOps (a portmanteau of “development” and “operations”) is a software development method that stresses communication, collaboration, integration, automation, and measurement of cooperation between software developers and other information-technology (IT) professionals.

That’s right, DevOps emphasizes communication between software developers and other information-technology professionals.

I’m going to take a bit of a harsh segue here to discuss what I do everyday at work. My official title is Programmer/Analyst, I am member of an internal Development team within my company that writes line of business applications for our internal customers. We automate repetitive tasks and write tools to make our company more efficient, freeing our human labor to focus on getting work done instead of clicking through websites repeatedly to get information or manually identify pieces of paper that should be attached to an account.

I want to emphasize two phrases from this:

Our internal customers: I see a lot in IT and internal software development to use the term “user”. This word is accurate, but it can often have a certain amount of animosity or complacency around it. We often see ourselves as keepers of the castle, and we allow “users” to use the services we provide.

This, I feel, is why IT is often seen as a cost-center instead of an important part of an organization, not because it doesn’t produce revenue. However, internal development teams have an almost identical issue; but it’s toward the IT team instead. There is often a lot of animosity between Systems and Development, we often see “the wall” and how bits are tossed back in forth. It’s been turned into comic after comic, and joke after joke.

DevOps is a practice that stresses communication between development and the systems, but I don’t think that definition goes far enough.

Automate repetitive tasks…focus on getting work done: My team’s slogan is “Automate. Create. Innovate”. In our industry there’s a lot of tedious work, a large chunk of our time is spent removing it or making it quicker. We create tools for our customers to increase productivity by letting them focus on their jobs, not ancillary tasks that just waste their time.

I think this is where my views about the term “DevOps” diverge from the mainstream. My team is responsible for automating repetitive tasks for the rest of the company, I feel where “DevOps” is concerned that means our role extends to our Systems team as well.

Piecing these two important details together, I believe a successful DevOps implementation does not involve making your Systems team learn to program, in tandem it doesn’t mean your entire development staff will ever replace your Systems team. Instead, I see it as a collaboration of your Development and Systems staff.

Essentially, your Systems team should become another customer of your Development staff, being provided with tools to automate repetitive tasks so they can focus on things that cannot be automated (and there is still plenty out there). Importantly, the relationship of customer and provider should be one of mutual respect. A company providing a service will not stay in business for long if they do not listen and respond to the needs of their customers, and a customer develops a certain amount of trust with a provider that engages with them to develop the products they use every day.

Don’t make your Systems team learn to program, find a developer or two who wants to learn parts of what your Systems team does, and then set them on a project to write tools to let them work more efficiently. Break down the wall between your two teams by establishing a relationship of mutual trust, where both teams can rely upon the other, and ask each other for the tools they need to accomplish their respective goals.

Integrating ember-cli with Maven

Recently I have begun working on a project that will have a ember frontend (using ember-cli) and a Spring MVC REST API for the backend. ember-cli handles the frontend build entirely using broccoli and while I could reinvent the wheel it makes more sense to use the build pipeline already setup by the ember-cli team. As such, I need a way to integrate the frontend build into my maven build so that my frontend can be easily integrated with my backend for packaging and deployment.

The following pom.xml is identical (aside from coordinates) to the one I am currently using in this project, it utilizes the excellent frontend maven plugin to handle the installation of nodejs + npm and calling out to ember-cli to perform the build.

Those familiar with maven should be able to pretty easily tell what this does, I’ll probably get around to posting a detailed explanation later but this was useful enough to share.

Chromecast Developer Terms prevent Windows Phone developers from working with the device

I am leaving for a business trip on Sunday, and after getting tired of crappy cable TV at hotels I decided to pick up a Chromecast to use while I am out of town.

I own a Nexus 5 in addition to a Lumia Icon, so I figured I could get some normal use out of it with Netflix and Pandora and then see if I could get it working on Windows Phone (for which Google supplies no official SDK).

To develop a Chromecast application you must register with the Chromecast developer program for a nominal $5 entry fee, then you must register your application and the URL the device will open when casting is initiated (there is no way to send an arbitrary URL to the device, it can only open a registered application’s URL). In my haste, I had signed up for this program and skipped over a very important part of the terms of service (emphasis mine):

3.2   You may not use the Google Cast SDK, either directly or by using the contents of the Google Cast SDK, to develop a standalone technology and/or to block or otherwise adversely impact any functionality of any Google Cast Receiver. For example, you may not build functionality equivalent to the APIs provided by the Google Cast SDK. You further agree that you will not create any exploits of any Google Cast Receiver, such as rooting a device.

Since you MUST sign up with this program to develop a Google Cast Ready application, there is no way (legally) to do so on a platform that Google does not officially provide an SDK for. Effectively, Google is blatantly saying “If you don’t use Android, iOS or Google Chrome, you can go away”.

After Google’s recent olive branch with the release of the Google APIs Client Library for .NET I am quite surprised that this language is still in the Chromecast Terms.

Gradle Installer for Windows

Gradle is distributed as a standalone ZIP file, which while adequate is not friendly to system management software and if you’re like me often ends up in the creation of a “Random Apps” directory in the root of their C:\ drive to keep them organized.

Just to make my life easier I’ve built a MSI installer for the current build of gradle, available at https://bitbucket.org/lithiumpc/gradle-win-installer/downloads

Kerberos Authentication with Load-Balanced IIS

Introduction

Using Kerberos (Windows) Authentication with a single IIS server hosting your application is easy! You just stand up your application and tell people to go to MYIISSRV/MyApp/ and it just works, simple! Unfortunately, Kerberos authentication becomes significantly more complicated when you wish to start load-balancing your application (in our example using IIS ARR’s Web Farm functionality, but the same would apply to any hardware or software load-balancer).

The Problem

When using Kerberos authentication in your web browser it essentially works like this:

1. 1. Locate SPN: Windows will search for a Kerberos SPN matching the host you are navigating to, for example: int-apps.mycompany.local will have a HTTP/int-apps.mycomany.local in Active Directory.
2. 2. Get and send ticket: Once Windows finds the appropriate SPN, it will ask the KDC (which in Active Directory is the Domain Controller) to encrypt a ticket for that SPN. Then your web browser will send this encrypted ticket back to the server to authorize you.
3. 3. Server decrypts ticket and verifies identity: Once the server gets your encrypted ticket, it uses the key matching the SPN to decrypt the ticket and verifies its authenticity. Assuming it hasn’t been tampered with your identity has been proven.

Again, this is really simple when you have one IIS server hosting your application, but there are a few gotcha’s that hit you once you start load balancing your app:

1. SPN’s are assigned to specific accounts. Every computer account in Active Directory has a set of default SPN’s assigned to it (HOST/NETBIOS-NAME and HOST/domain-name.mycompany.local along with a few others).
2. There can be NO DUPLICATE SPN’s (so two accounts cannot have a HTTP/int-apps.mycompany.local)
3. If the account your application pool is running as doesn’t have access to the key for the appropriate SPN, it cannot decrypt the kerberos ticket sent to it.
4. The way Windows looks up SPN’s can vary depending on what type of DNS records were used to resolve a given address.

Essentially, the above boils down to this: You cannot give the HTTP/int-apps.mycompany.local SPN to multiple machine accounts, but you need multiple machines to have access to this SPN because it is what the kerberos ticket will be encrypted with.

We need a way to share SPN’s between multiple machines, but how?

Identity Crisis

Oddly enough, SPN’s can be assigned to more than just machine accounts, they can also be assigned to any normal user account in Active Directory. So instead of trying to make duplicate SPN’s for each machine, let’s do the following:

1. Create a service account that will be used for your IIS Application Pools
2. Assign that service account the SPN’s you will use to decrypt kerberos tickets for any domains you will be accessing your server farm by.

Create the service account

I assume you are familiar enough with Active Directory to create a service account, and since this is not an Active DIrectory tutorial I will not be giving your a step by step on doing this, but I will state that an account should be created with the following settings:

Password Never Expires
User Cannot Change Password
Disable Interactive Login

Create the SPNs

For the most part Windows and various Microsoft Products create their own SPN’s where needed or they just piggy-back on the HOST SPN that is automatically created for each machine. Unfortunately, since we need to create some new ones we will need to polish off our command line skills (which shouldn’t be that rusty, everyone is using PowerShell now, right?!).

Use the following command to create a SPN for int-apps.mycompany.local and assign it to an account named “webservices”

setspn -U -S HTTP/int-apps.mycompany.local webservices

This will allow your service account to decrypt any kerberos tickets sent to it when users navigate to int-apps.mycompany.local.

Configure Application Pools

If needed, create a new application pool in IIS for any applications that will be load balanced and need access to this SPN. If you already have an application pool created for these applications you can simply modify it.

Right Click the application pool in the IIS Manager and select “Advanced Settings”

Click in the Identity field, then press the “…” button to open the Application Pool Identity window.

Select “Custom Account”, then click Set. Enter the credentials for the service account you entered earlier.

Repeat these steps for every server that will be hosting your applications. This does NOT need to be done on your load balancer since it just passes traffic right through unless you are trying to do some authentication offloading (my advice, DON’T).

Configure Applications

The first step in configuring your individual applications is to have it use the application pool you just configured. Right click your application in the IIS Manager and select “Manage > Advanced Settings”

Click in the Application Pool field, then click the “…” button.

Select the Application Pool you just configured.

Great, we are halfway there! Next, with your application selected in the IIS Manager’s Sidebar click on the “Authentication” section in the grid of icons.

Click on “Windows Authentication”, then “Enable”

Next, click on Providers on the right sidebar and enure only “Negotiate” is visible. If you want to allow NTLM as well that is your choice.

Lastly, return back to the application’s node on the IIS Manager’s Sidebar, then click on Configuration Editor.

Select the system.webServer/security/authentication/windowsAuthentication Section.

Ensure that useAppPoolCredentials and useKernelMode are both set to true.

Holy crap, are you tired yet? That’s it, you’re done! Almost!

Almost?

There’s always a catch! Here are some issues we have run into when setting this up ourselves.

DNS: Remember how the list of gotcha’s included “The way Windows looks up SPN’s can vary depending on what type of DNS records were used to resolve a given address.”

When using a CNAME to point int-apps.mycompany.local to lb01.mycompany.local Windows will attempt to use the SPN for lb01.mycompany.local, this is obviously not what you want. Use A records to point to your load balancer and Windows will use the desired SPN.

Accessing Servers Directly: By making this change the identity your application is running under no longer has access to the keytab of the machine account, so it will not be able to decrypt kerberos tickets when you attempt to address a specific server by its NETBIOS name or FQDN.

You may be tempted to do the following:

setspn -U -S HTTP/www01.mycompany.local webservices

But avoid the temptation, this will cause ALL HTTP services on the system not running under your webservices account to fail to decrypt kerberos tickets, a huge side-effect of this is WinRM and Powershell remoting stop working.

Instead, if you must access servers directly add another alias for them in DNS (remember, use an A record) and assign an SPN for that alias to your service account. This will prevent your need to load balance from mucking up management tools.

Questions? Comments? Please let me know.

Simple LDAP Authentication & Authorization in Ruby

For an application I’m currently writing I find myself needing to authenticate against my company Active Directory domain, now there’s a million authentication solutions out there for Rails but they’re all overkill and I’m using Ramaze for this anyway, so I decided to implement my own with Net::LDAP. I’m going to put the gist right here, and I’ll go over it in a moment

Now the code here is somewhat domain specific, but it’s fairly easy to adapt to other situations. Most directory configurations aren’t going to be simple enough where you can insert the username into a string to get the DN for binding (for example, my username is ‘snuxoll’ but my DN in Active Directory is CN=Stefan Nuxoll,CN=Users,DC=corp,DC=lithiumpc,DC=com) so you will need to search your directory to get the appropriate DN to authenticate against.

So let me go over some extremely basic usage

First we need to use find_user to retrieve the entry from the directory belonging to the user. From here we can call Net::LDAP::Entry#dn to get the distinguished name for the user (e.g. CN=Stefan Nuxoll,CN=Users,DC=corp,DC=lithiumpc,DC=com), after that we will use authenticate_user to try to bind as the user with the specified password, lastly user_authorized? will check to see if the user belongs to the appropriate security group.

I’m sure this is probably as clear as mud to some people, I’m not really awake right now, but I wanted to share this code before I forget as some may find it useful.

Hosting Your Own Blog = Nightmare

That’s it, I’m done. Hosting your own blog is a friggin’ nightmare, between needing to manage security updates for your software, making sure your CMS processes haven’t suddenly started hammering your VPS’ disk out of the blue (true story, go drupal) and then just dealing with managing changes to your web server config is too much.

Hi Tumblr, let’s do this thing.