Finding your way in IT: What type of sysadmin will you be?

I’ve been working with computers professionally for nearly 20 years and have been writing about Hyper-V for about five of them. My career has followed a circuitous path that has twisted through the rigid confines of by-the-book turnkey deployments and turned through the creative greenfields of system design and even stopped to polish the cobblestones of data analysis. I’ve had the pleasure of meeting some of the most talented and brilliant minds in the industry, but I’ve also been tortured by the presence of the think-they-know-it-alls and the let-it-go-to-their-heads. The IT industry is deep and wide, vast and fathomless, and it is growing all the time.

Writing publicly has introduced me to a much larger range of people, and therefore situations, than I ever would have encountered on my own, and I believe that I have a much better feel for what goes on in the world than I did before I started this particular phase of my career. I’m a bit disturbed by what I perceive as a trend: people that just don’t seem to have much interest in the field that they’ve chosen to work in. I am really disappointed when I see anyone decide to cut themselves off from entire categories of tools for some arbitrary reason. The attitude appears in many forms, but where I can really see it is in discussions around PowerShell. Just a few of the epithets I’ve seen thrown at PowerShell are: “archaic”, “yesteryear”, “downgrade”, “primitive”, “throwback”, and “the old way”. Of course, it isn’t just PowerShell.

I’ll admit that my natural reaction when I see comments like that is negatively judgmental. I tend to instantly stop aiding anyone that would rather suffer with a problem than even be asked to open a PowerShell prompt. I feel like I’m dealing with a hungry toddler that missed naptime instead of a professional peer. Sure, no one enjoys stepping outside of their comfort zone, but these things are just part of computing. If you won’t do them, why did you choose to get into computing?

But, I’m going to try not to become the grumpy “Get off my prompt” old man. I’m going to implore you to examine your options and decide where you want to end up and find your place in IT.

I envision three generic “tiers”, if you will, of systems administration.

The Automaton

The Automaton is the lowest rung, where we all start. The Automaton does things strictly by rote. The beginners do so because they don’t yet know any better and they’re still learning. Automatons typically work from scripts that someone else wrote and many issues that they are presented with wind up being escalated to someone else.

Automatons are a necessary component of the IT world. It’s hard to do well without going through this phase because book learning can only take you so far. Becoming skilled at fixing computers requires that you spend time fixing computers. Additionally, those in the upper tiers rely on a solid base of Automatons so that they can focus on other things.

If you’re at the Automaton level, the question is, do you want to stay there? Some people get permanently and unhappily stuck in the Automaton rut through their own actions. Here’s what they look like:

• Their first and best answer to every problem is “Uninstall and reinstall.”
• Their second and final answer to every problem is “Reformat the system.”
• Could probably be replaced by a well-written script
• Could definitely be replaced by someone that wants the job

The Mechanic

Mechanics are fixers. They have enough experience and intelligence to apply solid solutions to problems. When something isn’t going well, these are the men and women that set things back to right. Escalations with Automatons go to the Mechanics, and in all truth, most people that call in to the help desk wish that they could just talk to a Mechanic.

Mechanics are absolutely vital because they are the ones that the world depends upon to return an ailing system to its functioning state. As with Automatons, some get to this level and just stop. Stopping here isn’t all bad; some people are truly gifted at interacting with frustrated and uneducated users, which is critical to the mechanic’s job. It’s only bad when a mechanic goes stale. A Mechanic who has lost interest will often:

• Blindly apply things that have worked in the past, even if it’s been proven that things have changed
• Devolve toward the “Oh, just reformat it” attitude of the bad Automatons
• Answer many questions with, “We can’t do that” without giving the question much thought
• Avoid new or unfamiliar solutions, even if superior to the old

The Engineer

The Engineer designs the layout of the systems that everyone else uses. Unlike the Automaton and Mechanic roles, there is a great deal of what is commonly described as “art” involved in the role of Engineer. Truthfully, it isn’t so much art. With sufficient experience and attention, you gain what appears to others as a sort of sixth sense. You know when something will or won’t work, largely because you’ve seen it, or something very like it, succeed or fail. You can look at a vendor specification sheet and spot the numbers that they fudged, but mainly because you have experience with the ways that vendors fudge numbers. You ask questions that others view as visionary, deep, and insightful, mostly because you’ve suffered the consequences of not asking those questions. What really sets the Engineer apart from the Mechanic is not just seeing things as they were or as they are, but as they could be.

I know that a lot of engineers in more rigorous fields really resent that we even use the designation of “engineer” in our field because they are required to endure substantially more intensive training and examination than we do in order to be called an “engineer”. While I can certainly respect the effort that they put into their educations and certifications and licensing, this word is still the best analog for what we do. After years of doing follow-up work behind some very poor systems engineers, I sometimes wish that “[Computer] Systems Engineer” would become a protected title and require some of that intensive training and rigorous examination. It just isn’t realistic in our field.

It’s true that fewer people reach Engineer level than Automaton or Mechanic and that a great many wash out fairly quickly. However, there’s really nothing mystical about it, and Engineers can go rotten just as easily as anyone else. You know an Engineer has gone past his or her expiration date when:

• “Because I say so” and “because I’ve always done it that way” are the sort of answers they give to “why”
• Sincerely says, “Nobody ever got fired for buying X” and only ever buys X without ever even looking at Y or Z
• Has a tendency to become aggressive when challenged on technical matters — This is not to be confused with the Engineer (or even the Mechanic or Automaton) that is constantly being asked the exact same question, especially in those times when the intent of the question is to trip up the Engineer.
• Unwilling to mentor
• Has a lot of “always” and “never” design approaches. Examples especially relevant to this blog: “I always use fixed VHDX,” and “I never use Dynamic Memory.”
• Dismissive of new technology and paradigms — This is not to be confused with the cautious attitude. Anyone with enough experience to truly be called an Engineer has seen a lot of hot new technology burn out after a couple of years and a lot of promises go unfulfilled. For example, I think we’re in the 9th consecutive “Year of the Linux Desktop,” and I still don’t see the masses switching to Linux desktops. Worry when an Engineer commonly says things like, “Oh, I read an article on that Azure stuff. It’s stupid and we’ll never do that.”

Making Your Decision

Everyone should take a hard look at this list and do a lot of self-reflection. Expand upon my ideas — what other positive or negative traits belong to these various groups? Then decide what you want to be — decide who you want to look at in the mirror and who you want other people to see. Then decide how much effort you’re willing to put in to get there. If you’re not willing to put much effort into it, then it might just be that you’re not as interested as you thought. It’s also possible that you the title without the work. A lot of people attain that, too, which is unfortunate for the industry. Please, for the good of the world, find the level that you’re comfortable with doing well, do the best you can to be the best at that level, and stay there. If you’re not comfortable with any of it, switch to a more fulfilling career. Everyone will be happier.

How to Move Up

Since you read this far, I’m going to assume that you want to either improve at the level you’re at or move up. As a token of my appreciation for sticking with me, I promise to do my best not to disappoint. Here’s my pointers:

• Realize that the three categories I listed above are fluid. I separated those groups to make a point, but very few people fit precisely into any one of those categories, and no one can do it every single day. Many Automatons experience flashes of insight or inspiration where they do work that is advanced for their level. Every Engineer has “those days” where they just follow their daily runbooks and do nothing else.
• Love it or leave it. If technology does not excite you, get out now. To some degree, I think it’s normal that a person isn’t excited over all technology, but if this is all “just a job”, then you have made a poor career choice. Find the Help Wanted ads for the area that you want to live in, find out what sort of jobs they have that align with what you want to do, and go be that.
• Keep your chin up. Have you slipped because you’re in a miserable job? I get that. Once, when I was doing telephone support, my computer crashed near the end of a call. I wrapped up with my customer while the computer was rebooting and when it came back up, I entered in the call notes. The whole process took four minutes longer than the normal allotted time between calls (it was a 486, OK?). When I turned my phone back to ready mode, I had a voice-mailed reprimand from my supervisor because I neglected to tell her that I was taking a break. She could have called me directly and I would have answered. She sat no more than 10 feet away and could have just stood up to see that I hadn’t taken any break. I was unable to get the reprimand reversed. That wasn’t even the worst day of my job there. It was just the day that I started planning my exit. A lot of people around me just became blank-eyed Automatons. The moral of this story: I know that many of you work in horrible jobs. Make yourself better so that you can get a better job.
• Are you in the right silo? “IT” is such a short label that it belies the breadth of what constitutes “IT”. I have loved virtualization from the moment that a sales engineer made it clear to me what it really was. I hope that Hyper-V or some descendant of it lasts and that I get to work with it in some capacity and write about it until I retire because this is what I want to be when I grow up. Maybe that’s not for you. Maybe mail servers is your gift. You might be a database wizard. Maybe fixing and deploying technology isn’t your calling; the world has a permanent shortage of talented technical trainers. Of course, this is all stuff that you can find in any technical college brochure. Did you know that you can get a job where you do nothing but work on those devices in hospitals that show a patient’s vitals? You can work all day on radio tags that are used to triangulate and track objects and personnel through WiFi fields! There is such a variety within IT that you can do things that you haven’t even heard of yet. Shop around!
• You don’t have to work there. This chains right off of the previous two bullet points. In many places in the world, the United States especially, the single employer from graduation to retirement is no longer the norm. IT workers routinely change jobs without stigma. Personally, I like the stability of long-term employment, but I know that I could fairly easily skip from contract to contract and probably never be unemployed any longer than I wanted to be. I could get back into training and travel the country — maybe even the world — if I wanted. The immediate benefit of contracting and other short duration work is that you’ll gain a lot of experience in a short amount of time. You just might discover that little niche of technology where you really belong. If where you are is unbearable, allow it to convince you to change where you are, not who you are.
• Learn on your own. I am not telling you to sacrifice your life. I am a huge advocate of no-work family time. I also don’t believe that it is possible to effectively or efficiently bury yourself in study of even a field that you love during all of your waking hours. But, you’re going to have to give up some of your “me” time. This goes hand-in-hand with “love it or leave it”. If you’re only willing to invest time into exploring technology during the hours that you’re being paid and you won’t even consider putting your own money in it, you’re in a rut and it’s time to move on.
• Accept new things. Are you actively avoiding familiarizing yourself with PowerShell? You are in a rut. You are avoiding your career. PowerShell is just one example.
• Accept that things that you’re not doing are perfectly logical and valid. I remember complaining about a question on my Windows 2000 Professional MCP exam that dealt with something along the lines of a woman using a computer set to the Korean language trying to write a Word document in French, or something of the sort. “That would never happen to me!” I protested. And, it hasn’t. But it happened to somebody, and the problem certainly mattered to that person. Once I accepted that, I was able to envision circumstances other than those that I would likely encounter. The immediate benefit was that as I studied, I found it easier to pay attention to the sections that weren’t relevant to what I was doing at the time. Over the course of my career, I have done a great many of the things that didn’t apply to me at the moment that I learned them, and I was that much further ahead of those who didn’t have the same foresight.
• Remove the magic. If computers are still a mystery to you, solve that immediately. I think this deserves its own discussion.

Removing the Magic

When I was thinking about writing this section, I’d envisioned taking you through an exercise that I was subjected to many years ago. Unfortunately, that seems to be impossible on modern systems. An earlier version of this article talked through it, which was a real let-down. Fortunately, a reader volunteered to help out and provided a great deal of the material that I had wanted to produce for you. That reader has asked to remain anonymous. I am forever indebted to his good will!

What I had to do was first write a simple program in assembly language. By simple, I mean really simple. It displayed the letter “a” onscreen and exited. An earlier version of this article included some code that I wrote mostly from memory. Reader-submitted code that essentially does the same follows:

.MODEL TINY
.CODE
ORG	100h				; deals with 80x86 COM format. 100H says that the machine code starts from address (offset) 100h in this segment.
						; effective address is CS:100H. For com format the offset is always 100H.
_Start:
PUSH CS					; for tiny memory model http://stackoverflow.com/questions/5364270/concept-of-mov-ax-cs-and-mov-ds-ax?rq=1
POP DS					; for tiny memory model 
MOV DX, OFFSET Output	; copy the offset of the Out variable to the DX register
MOV AH, 9				; indicate that we want to use DOS function 9 (show DS:DX onscreen)
INT 21h					; tell the OS about it
MOV AX, 4Ch				; indicate that we want to exit back to the OS
INT 21h					; tell the OS about it
.DATA
Output DB 'a$'			; declare variable "DB" as the letter "a"
END _Start				; run the program at _Start

If someone out there with a 16-bit assembler and an operating system that can run 16-bit programs would validate that for me, I’d be much appreciative.

Using the tools of the time, this was assembled into a COM file. I then had to open it up in a hex editor. What I saw there opened my eyes. Once I sorted out where everything went, I saw that what I had typed into the editor had been converted almost verbatim into a binary format. Take a look for yourself:

You can easily see the “a$” that I wrote into the program over in the character-mode dump at the far right. What else can you see? Do you notice the two hexadecimal “CD 21” entries? Those are the machine-language representations of the “INT 21h” lines that I’d typed. I had to learn the hexadecimal representation for all of the other entities as well; commands like MOV and register names like AX. If you look, you can match up most of what’s going on in the hex to the code that produced it. You already know what the “CD 21” sections match up to. Look at the “09” in the preceding bytes and look at the corresponding code line. Look at the following “B8 4C” and look at the corresponding code line. Assembly language used to be nothing more than a human-readable transcription of machine language.

For me, it was an experience something like “seeing the matrix”. I realized that, like the punch cards of really old computers, my program was just feeding these binary bits into the CPU in order and it was doing what it had been designed to do upon receiving that particular ordering of bits. If I told it to do a MOV, it did a MOV. It didn’t think about it. It didn’t stop to analyze whether or not that was a good idea. It just did it.

Operating systems, assemblers, and compilers work a lot differently today. What you see above is a real-mode program — intended to run directly against the hardware. The assembler that I used didn’t try to find clever ways to make my human-readable code into better machine code. While you can still program in assembly, it looks dramatically different and the assembler will find ways to make more efficient and/or safer code, so you can no longer easily get that 1:1 visibility of human-readable code vis-à-vis its machine code counterpart. Also, Windows doesn’t even allow most of those things, so you wind up substituting in less comprehensible calls into the Windows API. For teaching purposes, it’s a shame, really, because the more distance you have between what you did and what the assembler turned it into, the less comprehensible your actions are.

After that, I got an even bigger lesson by duplicating the code in a somewhat higher level language, C:

main() 
{
	printf("Everyone else always says, 'Hello world!' but I won't.");
}

I had to take something similar to that, compile it, and look at the hex output. As with the first assembly program, it was amazing how much what I had typed matched up with the binary that was produced. As a result of the previous exercise, I was also able to mentally “disassemble” the COM program from my C program into perfectly usable assembly language. Extrapolate from there into all of the other languages, and you start to see that every piece of code that has ever run, whether it was originally written in hex or assembled from assembly or compiled from C or C++ or interpreted from JavaScript or GWBASIC or JITted from Java or C#, all of it eventually has to wind up as binary and fed through a CPU. What’s really important to understand is that we are spoiled by our modern programming languages. They make it appear as though it is trivial to do something that literally cannot be done. For example, the following is all perfectly valid PowerShell:

$Part1 = 'Here is the first part, '
$Part2 = ' here is the second part.'
$Whole = $Part1 + $Part2
$Whole
$SecondWhole = @()
$SecondWhole += 'Line 1'
$SecondWhole += 'Line 2'
$SecondWhole

You can’t really add two strings together. You can’t dynamically add elements to an array. If you don’t believe me, try it in assembler or C. Under the covers, PowerShell is doing something like creating new variables out of the old and destroying the originals to make it look like the impossible happened. Over the years, the folks at Microsoft have gotten really smart about building higher-level compilers and interpreters that can take these impossible actions and quickly rebuild them into efficient machine code.

I assume that most of you that read my articles aren’t programmers and don’t intend to be programmers, so you might be wondering why this matters to you. It matters in that even if you never program, and even if you never remember that DOS interrupt 21 function 9 displayed the contents of DX:DS, you remember that computers just take the bit patterns that they are given and do exactly what they understand those bit patterns to mean. They are not smart, they do not think, they do not hold grudges, and, above all, they are not magic. Everything that a computer can do can be understood by a human. Not just the geniuses that figure out how to get microscopic chunks of silicon to respond to electrical symbols in a reliable fashion, but regular, ordinary, everyday humans.

Once you understand that, and I mean really understand that, you will become better at doing administrative work. Software behaviors that once seemed incomprehensible seem natural. Sure, PowerShell lets you keep adding parts of a string together until you get what you want, but it has to keep creating new variables to hold the old data and the new data. With big enough strings, that can cause a lot of CPU and memory to be consumed — a lot more than the final product alone would appear to justify.

I also encourage admins to try their hand at designing GUIs. PowerShell is not really conducive for that, at least not for what I have in mind. The Community and Express editions of C# and Visual Basic serve very well for it, though. Next time you think a GUI is bad, or you think that it’s silly that you have to use PowerShell for something, try to build the GUI that you think should exist. Make sure to put in all the checkboxes and radio boxes and text boxes and wire up the logic that ensures that a user can’t accidentally pick a non-functional or destructive setting, and make sure that you cover every possible permutation that any user might ever require. If you manage to make a decent GUI, share it with someone; maybe they’ll do the work to wire it up. But, before you do that, try to think of all the logic involved with that, too. Whatever you put in the GUI, logic still needs to be built behind it to turn user selections into reality. I see two possible outcomes from exercises like that:

1. We uncover people who are geniuses at GUI design that should change careers
2. You discover that designing a good GUI is hard and cut everyone that does it (or opts to release a PowerShell module instead) a little more slack

Your Future

Hopefully, you’ve learned something from all of this, or at least started thinking a little harder about the world of IT and where you fit in it. Some parts of computing can be hard, but every career has that. If you’ve encountered something in this field that you just don’t want to do (like PowerShell), then the proper action is to move yourself toward something where even the hard parts don’t bother you. The pace and direction of IT are not going to shift or stop based on our desires that some things might be different or simpler. We must adapt or face the consequences.