Operability 2015

Such a good conference. Hand picked speakers, single track, good food, people, atmosphere etc. There were about 16 great talks but all these notes come from two outstanding ones.

These are the nuggets I could gleam:

Andrew Clay Shafer – a general rant

  • Broken gets fixed or goes away but shitty lives forever

    If you’re going to build something quick and hacky, make it an MVP style approach and throw it away when you’re done. If you build something shit, you are just going to live with the pain forever.

    So build things properly. Or build things to do one thing well, so if you try and make it do something else it breaks. Then it gets fixed. Just don’t do a shit job of something or it will prey on your dreams and make you eat shit.

    You pay interest on tech debt. Get rid of it, don’t live with it.

    Instead of pushing a solution harder and harder, try and figure out what the problem is. There’s probably something in between your problem and solution that needs to be solved.

    Einstein said something like “if I had 1 hour to solve a problem I’d spend 55 minutes thinking about the problem and 5 minutes thinking about solutions”. Do more hammock time, think about the actual problem and not the solution. Otherwise you end up with things in between your solution and your problem as I said in the last point. You’re shoe horning the wrong solution.

  • Learn how to speak and write if you want the best career in programming

    Most of our problems are socio-technological. The tools aren’t terrible, a better tool doesn’t solve all of our problems. We need to communicate better. We need to stop the tribal bandwagon behaviour of adopting cool sounding silver bullets. No. We need to work better as teams. We need to express our ideas coherently and spread good ones and eschew shitty ones.

  • Good operations is a huge competitive advantage

    Just look at Amazon. Being able to scale quickly, reliably, cheaply. Tightening the feedback loop to live. A deep and broad understanding of your systems and infrastructure. All important for going fast and reliably.

  • Principles >> Tools >> Techniques

    The principles come first. Don’t do it the other way around. Agile is a set of principles, apply them differently in different situations. Don’t focus on the tools that you’re familiar with, focus on the principles and pick the best tool for the job.

    Shafer coins “organisational learn” – how quickly an organisation can pick up a new principle and apply it. When they understand the principle they will pick the best tool for the job. Teach the principle not the tool.

Charity Majors on devops hiring, culture and key traits

  • A good interview
    • Interview should be done by someone with a good understanding of what the day-to-day will be for the applicant.
    • Questions are leading and broad. Questions that have more than one answer. Then the candidate has a chance to show off what they know, and not how well they can answer a specific question.
    • Similarly, tease out the breadth versus depth of their knowledge. Are you looking for an expert in MySQL? Then test the depth of their knowledge of MySQL. Are you looking for a good all rounder, good at monitoring, logging, site reliability, databases, config management and AWS? Then test their breadth of knowledge with broader questions. Bear in mind that everyone has weaknesses, so if someone doesn’t know much about database replication, try and talk about something they do know about.
    • Put people at ease. Let them solve the question in the way they want to. If they want to put headphones on and get into the zone for 15 minutes and google stuff then let them do that, and ask them to explain their work afterwards. Try not to shoe horn people into a rigid interview process. But also be conscious of your process, you can’t be lazy and overly relaxed, you need to engage and adapt the process per interview, according to the person.
    • Good references are worth a lot in this industry. If someone you trust has vouched for this person, that’s worth more than a CV.
    • Some good interview questions: What is the biggest disaster you have experience and or caused? Can you keep your sense of humour under pressure? How do you feel about your former job and former colleagues? If they complain ask them about how they tried to change the situation. Talking about how they solved a real world problem is more revealing than watching them code an arbitrary task in real time. Ask them about a problem they have solved previously, from high level down to the details where you see fit. What is your favourite problem set to solve?
    • We look for reasons to say no to a candidate, and not reasons to say yes. Imagine sitting on the other side of the table. You want to be asked lots of leading questions that give you a chance to show off your knowledge, your enthusiasm and your personality. So make the questions match up to that. Broad, leading and with many correct answers. Then differentiate candidates based on how much better they are not on how much worse.


  • A good ops engineer

    Taken verbatim from slides:

    • Strong automation instincts
    • Ownership over their systems
    • Strong opinions, weakly held
    • Simplify, simplify, simplify
    • Excellent communication skills, calm in a crisis
    • They value process
    • Empathy
  • A good ops culture

    The behaviours you call out and reward are the behaviours that get repeated. Do not reward people who work overtime and don’t take holiday. You know that it’s good to work normal hours and take regular holiday. So reward other behaviours. Behaviours like: giving good feedback to other team members, integrating well with another team, perservering with a difficult project, writing good documentation, etc.

  • A good ops engineer at big company
    • structured roadmap
    • executes well on small, coherent slices
    • classical CS backgrounds
    • values cleanliness & correctness
    • technical depth
  • A good engineer at start up
    • comfortable with chaos
    • knows when to solve 80% and move on
    • total responsibility for outcomes
    • good judgement
    • highly reactive
    • technical breadth
  • How to spot a bad ops engineer
    • tweaking indefinitely & pointlessly
    • walling off prod from developers
    • adding complexity
    • won’t admit they don’t know things
    • disconnected from customer experience
  • How to lose a good ops engineer
    • all of the responsibility, none of the authority
    • all the tedious shitwork
    • blameful culture
    • no interesting operational problems

These two talks really resonated with me, so I took a shit ton of notes. But there were loads of great talks and many great ideas.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s