Your Ad Here
 Wednesday, July 23, 2008

In the case of Terry Childs, a network admin who gained notoriety recently for locking the City of San Francisco and his managers out of their own critical network, comic-book style progress has been made, with Childs' attorney inviting the mayor of SF to a secret meeting at the jail, where Childs handed over the passwords he'd previously refused to disclose.

Childs' lawyer, again in typical comic book fashion, has also come out saying that Childs' actions were essentially noble and that he was acting to protect the network he built from his management and peers, whom he characterized as being neglectful and without the proper knowledge to support the network. About what you'd expect from a defense lawyer in a public case, I suppose.

But Childs is in no way a hero. Even if what he says is completely true, he's (allegedly) committed a real crime. He does not own that network even if he helped build it, and regardless of whether the management in his department was capable of exercising its responsibilities, when Childs locked everyone out he crossed a clear line. If it was to make a point, he simply went overboard. The whole unfortunate case just smacks of ego and manic behavior.

But from arm's length the city doesn't exactly look like a helpless victim, either. Any professional management team that creates an environment where one person can control a critical and sensitive network in the manner exercised in this case has missed some of the most crucial and common-sense aspects of IT and security design. In fact, most of the time when cases of one-man-too-much-power crop up, we find that the IT staff is also responsible for security with little or no separation of duties, no checks and balances, and no controls to ensure one bad apple doesn't ruin the whole barrel.

Was Childs right? Absolutely not. Was the City wrong? I don't see how you can argue otherwise.

You'd likely be surprised how many real-world computer networks - big and small, important and less so - are run on the concept of "we just trust that one guy." It's what we call a "Beer Truck" risk problem: If I'm that guy you trust, what if I get hit by a beer truck and killed, or alternatively what if I drink everything on that beer truck and go nuts and wipe out the network? What then?

Systems should be set up to ensure no one person holds all the keys. Over the past few days I've read comments made about this story, in many cases by angry IT-types who say if you hire someone you have to give them access to everything and you have to trust them to do the right thing. Otherwise they cannot do their job, you're a terrible person and your network and systems are doomed. That premise is simply and blatantly false, and in fact following that method puts you in the same boat the City of San Francisco has just found itself in. Please, don't listen to the old-skool IT admin crowd, telling you to hand it all over to them because you obviously don't know what you're doing. Fire those guys and find some real help.

If you want a healthier view of the situation, check out articles written by smart, thoughtful people, like this one by Paul Doyle. Also, Paul Venezia wrote an in-depth article about what went wrong, with some detailed inside information.

To be clear, no one person should control all the systems. Control and authority are not the same thing. Checks and balances are important. The Air Force doesn't allow one person to perform all the steps needed to launch a ballistic missile, right? Apply the same principles to your IT systems.

Case in point: I was the chief security executive at a major online financial services company. I had administrative access to nothing. I couldn't even get in the data center without an escort and records being kept. I had no account access to critical or sensitive systems. And no one person there could make changes in a vacuum. IT workers didn't have access to security systems. Security workers didn't have administrative access to anything by default. And we operated effectively, smoothly, with full knowledge of what was happening on the network and systems. No one person had control. Authority, sure. But actual control of systems? No. To operate otherwise would have been negligent.

I often preach the value of formalizing security management and putting proper process, technology and organization in place to ensure a good, stable system that can effectively support business. One of the pillars of an effective security management system is hiring good people (probably not ones who have been convicted of aggravated robbery in the past, sorry) and separating duties in a way that protects everyone involved - employees included. Doing so is not punishment, it's just good common sense.

If nothing else, lets hope businesses and governments all over learn from this embarrassing public spectacle. There are standards out there (my background and experience is in ISO 27001, an international security management standard), the very purpose of which is to make sure things like this don't happen. It's high time to start using them.



Add/Read: Comments [6]
IT Security | Tech
Wednesday, July 23, 2008 12:04:17 PM (Pacific Daylight Time, UTC-07:00)
#  
Wednesday, July 23, 2008 12:33:04 PM (Pacific Daylight Time, UTC-07:00)
Remember the day the CEO sent an email to everyone in company listing his favorite porn sites? Gotta love outlook viruses.
Dave
Wednesday, July 23, 2008 1:58:43 PM (Pacific Daylight Time, UTC-07:00)
Before you say that this individual did not own the network and did not carry ultimate responsibility for it, how much would you place on a bet that if the network were to massively suffer him, or someone like him, would eventually be trotted forth as the scapegoat?

I'm not saying what he did was right - it wasn't. I do believe that the flip side of the situation should be examined. If he doesn't own the network and isn't responsible for his coworkers ability or behavior, then he shouldn't be held responsible for those same coworkers when they foul up and things go awry. Prior to his actions it sounds like he would be the guy who was held responsible, regardless of their origin.

In terms of having models and access devices in place the city should examine when their administrators were last allowed to leave the building without a pager or cell phone, or take a vacation without said devices. You can't demand access to an individual 24/7 and not give him methods of accessing what he administrates. To do so is asinine.

I know I've been in many situations in the past where I was held responsible for just such actions.

*shrug*
Brian S
Wednesday, July 23, 2008 2:12:00 PM (Pacific Daylight Time, UTC-07:00)
Hi Brian S -

Yes, he most certainly would have been held responsible for problems on the network, but he doesn't own it. I have also been in similar situations, and no doubt this network admin was between a rock and a hard place. But his choices were still poor ones. Read the Paul Venezia article linked above. I've known several network admins and engineers with a similar, unhealthy mentality. It's important to realize, I think, that in this case it's entirely possible that both Childs and his managers/employer were in the wrong.

It's not always the case, but as IT Pros we sometimes put ourselves in situations and create circumstances where we're the "heros" who the bosses can't afford to lose, and the subsequent scapegoating is a side-effect of that.

There's no perfect answer for every situation, but you have to admit that in this case he landed in jail for a reason. He might be a great admin with good, yet badly flawed, intentions. I'll give you that. But I won't excuse the apparent behavior or mistakes. That's all preventable, and in the end that's what matters.
Wednesday, July 23, 2008 2:52:42 PM (Pacific Daylight Time, UTC-07:00)
Greg,

I couldn't agree with you more about Childs being in the wrong. As a 12 year IT admin, his actions are inexcusable. With that said, I've worked for employers who put the employee in the position where he has to "be the hero". It's part of the unwritten list of expectations from his manager who won't hire the necessary resources to proactively manage systems and security. Oftentimes, the manager is in the untenable position of having to get everything done with no budget, and that is a problem with the management of the company.

Having worked in companies who didn't believe in the value of IT, it is obvious that the City of SF does not value IT and is not willing to invest in appropriate procedures and individuals to create a mature IT environment. It appears that Childs worked there over several years. I doubt he came to the conclusion in his first day or year that this is something he needed to do, but if you're beaten down consistently, you eventually come to the decision to not trust your managers or co-workers. This is especially true true if you are held responsible for their actions or inaction. I've been there and it sucks. It is demoralizing and depressing to work the extra hours and give up much of your outside life only to be shown that your management doesn't care.

In the end, he should have taken the only true solution available to him ... leave. I could be wrong, but I believe that San Francisco still has a fair job market for IT professionals. Life is too short to work in a situation where you aren't valued and can't trust those you work for/with.

Jerry

Jerry C.
Wednesday, July 23, 2008 3:00:32 PM (Pacific Daylight Time, UTC-07:00)
HI Jerry - good to hear from you. :)

You said it well - thanks for your insight. From someone of your caliber it has real weight. I agree that middle managers are also often caught in a tough spot. Been there. But, as you allude to, when one allows himself to be victimized, one gets the consequences, as well. Of course I don't know any of the people involved, but we have all known those IT people who are technically excellent, yet are angry and very difficult to manage or work with. That's a personality that tends to build upon itself and cast blame. It's unfortunate, and IT managers need to be able and willing to recognize it and deal with it before it manifests itself in bad ways.

Just my two cents. :)
Thursday, July 24, 2008 3:22:31 PM (Pacific Daylight Time, UTC-07:00)
You're right. I've worked with my fair share of angry IT folks. Some I think are born that way, and others are created by the environments they work in. Some get a taste for power and do everything they can to remind people that they are important and have the keys to the kingdom. I worked with one guy at a former company who constantly threatened end users with reducing their internet bandwidth to 9600 baud. He was joking, sort of. Given the trust you place in an admin position, I agree with your assessment that it probably isn't the best move to hire someone in a position with that level of power who has a previous aggravated robbery conviction.

Angry IT people create their own perception problems. If you partner with the people who rely on you, they come to respect and rely on your advice and skills. If your attitude sucks, they'll do whatever it takes to go around you or circumvent the system, creating more frustration on the part of IT. It's a vicious cycle, and I've only seen it broken by someone from IT making a genuine offer to partner with the business. Unfortunately, partnering takes work. Lots of it. Most IT departments aren't funded at a level to make it possible which is even more unfortunate.

In the end, most IT people just want to come to work, be involved in the process, have some control over our fates, and accomplish something ... anything by the end of the day. :-D

Jerry C.
Name
E-mail
Home page

Comment (Some html is allowed: a@href@title, b, blockquote@cite, em, i, strike, strong, sub, super, u) where the @ means "attribute." For example, you can use <a href="" title=""> or <blockquote cite="Scott">.  

Live Comment Preview