Wednesday, December 3, 2014

Winter is Coming… And I’m fine with it

Today I read an article on TechRepublic about how robots and computers automate more and more roles previously carried out by humans, and how we won’t like what’s left.

We all know what’s coming, manual labor will be replaced by machines, leaving only the jobs that (still) require a human mind. I’m sure you cannot throw a stone without hitting a politician or spokesperson who will tell you that this is bad, but at the risk of coming off as a detached and condescending elitist, I will have to respectfully disagree.

Change Happens

Change happens. Always. This isn’t a new thing. Change has been happening since the beginning of time. Literally. Without change we’d still be a potentially explosive situation in the middle of nothing, because the Big Bang wouldn’t have happened.

Okay. I’ll agree that that was just some demagoguery. Forget everything that happened in the last couple of billions of years. Let’s limit the discussion to technological advances. Disruptive ones. Like agriculture. Advances were being made that changed the world, and the roles humans played in it. It started some 10,000 years ago, and is still going on today. Still not convinced? Still think this is demagoguery? Okay, I’ll put that aside too. Let’s talk “real” technological changes.

How about the Industrial Revolution? Those factories? Them machines sure replaced a lot of people didn’t they? Please note that that was going on some 250 years ago.

Change is a Good Thing

The Japanese word “Kaizen” literally translates to “Good Change”. Now, I’ll say straight up that not all change is good, hence the differentiation, but I believe that it is more often better than it is worse. I don’t have any scientific proof of it, but I seriously doubt that anyone could argue that their lot in life is actually worse than it would be if they were in their comparable station in life in or around 18th century Earth. I think that even people living in 3rd world countries all over the globe are as well of as they’d be, or better. I believe that just being able to discuss their issue, having first world people become aware of 3rd world people is an improvement. We, as a rule are becoming better people.

Please note, that the comparison I am making is between one person, and his or her own lot in a world “stuck” in the past. I believe that inequality, while it is a touchy subject, is irrelevant here. I know that there are people who hunger for bread living a few miles from people who have more money than they could ever spend. But was that not always the case? I believe that I myself, my friends, and probably anyone who has a computer, tablet or smartphone with which to read this post lives better than most nobles did a few centuries ago, and better than any king just a few centuries before that. And I’m just making a (fairly) decent living.

So What is the Problem?

Adapting to change is difficult. Disruptive change, more so. When something that you know, and believe in may become invalidated and obsolete, it may feel overwhelming and scary. Other times it just makes you feel irrelevant – just like your (now useless) knowledge.

People have been losing their jobs to machines since forever, possibly since ten thousand years ago, definitely in the past 250 years. But this isn’t a bad thing!

For every manual laborer whose skills were no longer necessary, two jobs showed up for factory line workers. Old jobs transformed to adapt to the new world. New jobs were created. Some people adapted, others did not. Those of us who adapted were better off for it. Those who didn’t, weren’t. And we can and should feel sorry for them, and that is okay, and it is okay to rejoice in our own improved lot, because the change made our lives better, as a rule, or it wouldn’t have happened.

The issue at hand, I believe is that change itself has changed!

Change is Changing!

Change was always happening, and the rate of change was always increasing. But where in the past change happened over billions of years, until it could be measured in millions of years, and human advancement was then measured in millennia, at some point changes became so great and so quick that we started to notice. The industrial revolution was, I believe, the first such point in time, with change so great and so fast that it one could perceive life before and after it, within one’s own lifetime. Since then the changes began to happen every several decades – factories, the telegraph, automobiles, airplanes, radio, television, computers…

Computers.

From the 1943 legendary statement, (allegedly) made by Thomas J Watson that “there is a world market for maybe five computers”, to a national average (US) of more than five (5.7) internet connected devices per household(!). In 70 years, computers have transformed (almost) everything that we do, allowing us to do more, much more, in less – much less – time. Everything. Including research work. This means that the rate of change itself has changed, dramatically and visibly increasing.

Why, up until roughly 20 years ago, hardly anybody outside colleges had any personal or commercial access to this overgrown, BBS-on-steroids thing called the internet. Ten years ago, I had a palm pilot, had a slow GPRS modem to connect to it, and was the only one I knew that had one such. Seven years ago, the first iPhone came out, and while there were “smart” phones before, it was the first device that was actually accessible for the general population.

Now? More than half of the entire world population have mobile phones. 31.1% of the population have internet access, meaning that just under a third of the people in the world have access to the sum total knowledge of human knowledge(!). Let’s ignore the fact that most people’s usage has nothing to do with knowledge (except in the biblical sense).

The Pros

Is this a good thing? I think it is. For me personally, at least. I cannot imagine being able to do my work, without access to millions of code samples, documentation, tools, guides, online books, etc. Twenty years ago I had to remember most of the software languages that I use, or else take a (heavy) book off the shelf and flip through what I couldn’t remember. Ten years ago I had intellisense and computers had enough memory and disk space for references to be stored on disk for quick access. Today I don’t ever bother to install the help, documentation and SDKs that come with the tools and languages that I use. It is quicker to Google it. Googling even became a word. Almost everybody knows it and does it (except for those who must Baidu it, and those who choose to Bing it, and those elitists who prefer to duck-duck-go it).

The Cons

Some people take exception at this change. From developers who claim that Microsoft-stack coders who rely on intellisense have become crippled by their inability to code without it, to my wife who gets upset because I do not (cannot?) remember anything that isn’t on my calendar, people believe that we who rely on the internet to remember for us have lost something by giving up the skill of memorization.

Personally, I think that a skill that is no longer needed is not worth having except for recreational purposes. I don’t remember how to sheer sheep, gut a fish or skin a rabbit, because I never knew how to do these things. I never needed to learn (though if I did, I’m sure I could Google it).

My main argument is that it is foolish to waste brain power in learning things that will soon become obsolete.

And that is our problem

Changing is Difficult. Not Changing is Fatal

Saw this quote somewhere. It is largely true, I believe. For myself, I love change. I can’t wait to see what the next big thing is, what’s over the horizon? What new technology will come out tomorrow…While I appreciate my own generation’s unique stand point, being born on the cutting edge, riding the wave of technological change, We are immigrants to this brave new world. I would love to have been born today, to be a true child of the millennium, to be born a native child of the world of technology. What does my eldest daughter at 13 feel, never knowing a world without the internet? How will my baby experience the world, growing up in a world where scientific and technological advances I can barely imaging will be common place?

But it is a constant struggle, riding the wave. One must adapt to change quickly, one must learn to devalue what one knows, in favor of what can understand and extrapolate. Hanging on to yesterday will cause you to fall into the chasm with it.

I figured that out on my own. My children will have that engrained in them. But many of my generation don’t understand that. Most of my father’s generation do not. How surprised are we to see our parents have a truly thriving online presence? Our grandparents? I learned to work with computers at an extremely young age, because I saw my father, one of very few at the time, using one, programming. My kids still perceive me as an expert on technology, though I already defer to my daughter in some areas (social networks, mostly).

My mom can operate a computer if told exactly what to do, step by step. I have friends whose mothers can read their email, because they have to, as part of their work, but go to pieces if things don’t work just right, because they fear the computer. Computers are alien to them, and the change they bring is beyond them.

My grandmother, and some uncles will still want to explain to me how to get somewhere, all while I politely explain that I just need the name of a place or an address and Google Maps will do for the rest.

That’s all okay, they have their ways of doing things, and I have my waze.

But not at the workplace.

At the office, at the factory, time is money, and anything that a computer can do, it can do faster and with greater precision than any human can cope to do.

And once a computer can do something, it is only a short matter of time before it will replace a human doing it.

What will such human do?

Learn to do something new, or become extinct, obsolete, unemployed.

The Way of Tomorrow Today

At any job interview, I will tell my would be employer that the biggest asset that I bring is adaptability, my ability to learn, knowing how to know. More than one customer asked me a question, and when he saw me searching the internet for the answer, he or she asked me something to the equivalent of “If you don’t know the answer why am I paying you”? My answer is always a simple statement that I’m paid because I know enough about my domain to Google faster than anyone he or she employs, I know where to search, I can understand what I read faster, and extrapolate a better solution to the problem than anyone else (perhaps a bit arrogant, but selling yourself short is never a good idea, and stammering any other answer would be worse).

What is my point? My point is this, in order to thrive in the world today, one must cultivate the skill of learning in one’s domain of expertise. One must learn to adapt to disruptive change, to seek out these changes and prepare, to look for opportunities to adapt first and come out on top.

That, or fade away. Or hope you can retire before becoming completely obsolete.

I consider myself lucky (though I believe in making my own luck), I could never as a kid sit down in one place at school and learn to retain knowledge I considered useless. I instead learned quickly on my feet, figuring things out in the middle of a test, extrapolating from whatever I learned before. Today this a prized skill.

Others who are less lucky have to learn now. Change will never stop. Not everybody will hold back, and not everybody will stop inventing new ways to replace human labor with computers. Perhaps one day it will all be computers and machines doing the work, and perhaps we can become a utopian society where machines can sustain comfortable human life for all, and people will simply do what ever they can to make themselves and the world better, not for monetary reward, but because they want to fulfill themselves (see – this isn’t communism).

One can hope.

And adapt.

Friday, October 3, 2014

10 Things You Should Know When Submitting a Bug

An on-call technical support engineer wakes up groggily at 2am, when his phone rings. It’s a department manager. “Huhhhh? Huhhlllo?” he mumbles into the phone.
“Your software doesn’t work!” states the angry voice on the other end.
“Uhhh… Could you be more explicit, sir?” asks the still bewildered techy, trying to figure out what’s going on.
“Your software doesn’t #&@<ing work!”
So, that might sound like a joke, but for many people in the software development business, this is just another workday. It seems that whenever somebody has a technical issue, they become even more technically illiterate than they might normally be. Personally, I believe that fear and frustration with their unknown situation, coupled with annoyance at having to deal with a less-than-perfect product reduces the end-user’s ability – perhaps at a subconscious level, perhaps not – to be a part of solving the issue. They want it solved, and they don’t care how.

We want information. INFORMATION!

No, I’m not the new number two. I’m just another developer, who has to solve problems with software, and consequentially, solve problems in the software that solves the problems. I get bug reports coming from end users, testers, middle managers – sometimes even fellow developers – that leave me without anything more than a general sense that there was a disturbance in the force, and the software is not performing within acceptable parameters.

Please understand! Most of us are actually eager to solve problems! A lot of us actually got into this business because we have a hacker mentality, we’re tinkers at heart, and we view every problem as a challenge to meet! Don’t hear us clicking away furiously at our keyboards, guzzling coffee, muttering curse words (often in foreign languages such as Russian or Klingon), and then stop with a resounding “WHOOP!” of joy? That’s a feature we developed or a bug we fixed.
Help us help you.
The items below are the top ten things every developer wants you to provide – as best you can – to help him or her understand what the problem is and how to reproduce the issue:

1 – What do you mean to say?

I know that for many of you, the official language might not be your native one. Where I live, roughly half of the people are foreign-born (which is great, by the way – I like diversity). It’s fine. But please, please, do your best to write your bug report in a precise, understandable manner:
  • Be specific – did you load a file? Fine. Did you click on a button, select from a menu, or did you use a keyboard shortcut? It might be important. For bonus points, try all different ways to do whatever you did. If it works one way and not the other, let the developer know.
  • Be explicit… as in precise. Instead of the ‘it’ pronoun, describe it (‘the window’, the ‘message box’ the ‘radio button’, the ‘phone’, etc. Just not ‘it’).  Use the Find command in your email client, browser, MS Word, or whatever you’re using to report the bug, search for the word ‘it’ and replace it.
    Also, avoid ‘they’, ‘he’, or ‘she’ unless there can be absolutely no doubt about the person’s identity.
  • Reread what you wrote – unless you’re completely incompetent as a writer in the language in which you’re writing (and I guess it’s okay, if they hired you despite your language deficiency), you should be able to see if you think that you were clear. If you weren’t, please correct your grammar. And for the love of your chosen deity, please use a spellchecker. Correct those squiggly red lines!

2 – Can I please have a bug report, hold the attitude?

Jokes aside, we usually do not insert bugs into the system intentionally. At best it was the one defect that slipped the developers’ scrutiny. Often it is the product of being rushed to meet a tight deadline. At worst it is the result of negligent work born of the despair of either not knowing how to work properly and worse – knowing how things should be done, but not being able to do them the right way (see aforementioned deadline).
Either the poor developer is mortified by the bug (extreme unhealthy reaction – but it happens), or the developer has become apathetic to the bug through continual disillusionment (extreme unhealthy reaction – especially for the organization that employs him or her), or anything in the middle.
There is no point on the reaction scale at which berating or verbally abusing the developer is productive. Or even acceptable.
So take a deep breath and stick to the facts, Mack!

3 – What exactly seems to be the problem?

imageNo, we don’t think that you’re making it up, that you’re dense, or that you’re bothering us with trivialities. We just want to know what kind of a bug you might be dealing with now:

  • Did you get a weird error message?
  • Did the application crash (i.e. suddenly shut down)?
  • Did the application freeze?
  • Does something look wrong?
  • Is something missing?

4 – Do you have any evidence?

Yes, for god’s sake, we believe you. We don’t think that you’re off your rocker, if you have one. We need whatever evidence you have to help us identify the problem, find its source, and resolve it. So if you have it, give it up:
  • Screenshot of the problem
  • Exact wording of the error message
  • Log file (if you don’t know what that is, don’t worry)
  • Any other kind of output?
Also, if you’re a tester, please come prepared – you’re working with software, our software, and you know it might malfunction.In fact, if it didn’t, you wouldn’t have a job, now, would you? Since that is the case, please have some kind of diagnostic tools installed to help you gather the evidence: loggers, video captures of your testing session, etc.

5 – Do you have any special set up or configuration?

Especially if you are a developer or tester, you might have a special configuration file with specific setting that you are using. Please provide any non-sensitive configuration, or at least describe it. Anything you know about how you use the software can shorten the time it takes to fix the problem.

6 – What did you expect to happen?

No, it’s not a cynical way of stating that that’s the way the cookie crumbles. We don’t always know what you believe the normal behavior should be. You may be right, you may be wrong. If you’re right, we need to fix it. If you’re wrong (i.e. the infamous “not a bug”, a.k.a. “by design” / “as designed”), we need to fix our documentation. So please let us know what you expect.

7 – What did you do before the problems started?

horrified pupLemme guess: You think that it is your fault. Well, it isn’t. Period. It never is. Get over yourself. Stop wallowing in self pity and shame. This is me being cynical and stating that that’s the way the cookie crumbles. Nothing but the simplest programs can ever be completely defect-free.
What we mean to ask is “Can you please tell what steps I need to take in order to reproduce this myself”?
So please, list the steps you took, as best you can, so that we can repeat those, as many times as necessary, until we find the problematic part of the code that needs to be fixed. It is usually something in the code that is responsible for handling the steps you took, so this information is vital!
This is where a video recording, coupled with a log file and input recording would be nice. Testers – please use those.

8 – It works fine on my machine.

Okay, this one goes out to all you developers out there. There is nothing our coworkers hate hearing from us more than this line. It is not okay. It is still your responsibility to resolve the problem. I once worked with a manager that suggested that any developer that shrugs a defect off with “it works fine on my machine”, will be shipped off to the customer with his machine, in order to solve the problem. Said customer was in another country, where summer temperatures reached over 50 degrees Celsius (122°F!).  To my knowledge the threat was never actually carried out, but the developers there used the phrase less than any other team I’ve worked with.
What we mean (at least what I mean), and what we should say is “It works fine on my machine. This means that the problem is probably related to your environment. Can you please describe the conditions in which you are running this software?”.
So please describe your execution environment (re: what your machine is like), to the best of your knowledge. If you don’t know it, don’t sweat it – but if you are a non-technical user, and can find out this information (Wikipedia is a good enough explanation. You don’t need a PhD to understand most of it), you’ll really make your developer’s day:
  • What version of our software do you have? Testers – you must know this. Devs – you must make this knowledge available!
  • What are you running this on (smartphone, tablet, laptop, PC, server, virtual machine)?
  • What operating system are you on – we might not know (Windows, Linux, Mac OSX, iOS, Android, Samsung? etc.)
  • What version of the operating system do you have (Windows XP, 7, 8, 8.1, Mac OS 10.6, 10.7, 10.8, 10.9, iOS 6/7/8, Android 2.2/2.3/4.0/4.1/4.2)?
  • What version of Office do you have (if any)? It may be relevant.
  • If you know the software to be a sub-module or extension of some other software, please let us know what version of the host software do you have (e.g. you’re running an extension on Visual Studio – is it 2010/2012/2013? Which update?)
  • If this is a web application, what browser are you using? What version?
  • How much memory does your device have?
  • How much free disk space does it have? All disks / volumes, please.
  • Screen size?

9 – Are there any specific conditions / times that this happens?

Sometimes a bug will consistently appear whenever the application is executed. In other cases it might be limited to happening only sporadically:

  • Does it happen every time you run it?
  • Only every other time? Third time?
  • Only mornings?
  • Every Sunday at 2am? This is not a joke. If I know that the nightly backup happens at that exact time, I can deduce that the two events are interacting and this causes the problem.
  • Does it happen only when some file is open in notepad?
  • Does it appear to be completely random?

10 – Is this happening to anyone else, as far as you know?

This one goes especially for corporate and enterprise users, as well as testers in teams. Nobody expects Joe Six-Pack to go knocking on doors in order to figure out if he’s special or part of something bigger.
By now, you probably know that I’m not singling you out as an idiot, or something, but rather want to understand what is happening. If this is happening to everyone on your floor (but nowhere else), it might be some network issue. If it is only you, it might be a configuration issue, but if everyone on your team who’s using Internet Explorer 10, as opposed to 8, we know that there’s a compatibility issue in the code. The developers need to be able to identifying what is common to those who have the problem, that sets them apart from those who do not have it. It’s called differential diagnosis.
See, every bit helps. If you’re on a team, you really could go that extra mile to see if your teammates have similar issues. Especially if the software is built by the same company that you’re working for. You’re all in this together.

In Conclusion

We’re all in this together – developers, testers, operations – we usually (should) have the same interest – to make sure that we create a better product. So please – pretty please – with relish(!): be professional.

Thanks in advance,
Assaf – a developer.

Monday, January 6, 2014

12 Revs of Software

Would you put your gifts under this?A Version from St. Service Bus

’Twas the week O’releasing, and all through the teams,
Not a programmer testing, at least so it seems.
The builds barely holding, the crashes abound,
In hopes that some luck, or a safe place be found...

[Enter the choir. You can sing along. You know you want to…]

The Twelve Revs of Software

BadSoftwareSnakeDMOn the first rev of software, my dev team dropped on me
A piece of software that I can't release.

On the second rev of software, my dev team dropped on me
Two breaking smoke tests,
And a piece of software that I can't release.

On the third rev of software, my dev team dropped on me
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.

On the fourth rev of software, my dev team dropped on me
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.

bug_report-250x250On the fifth rev of software, my dev team dropped on me
Five bugs regressing,
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.

On the sixth rev of software, my dev team dropped on me
Six missing specs,
Five bugs regressing,
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.

angry-bossOn the seventh rev of software, my dev team dropped on me
Seven bosses angry,
Six missing specs,
Five bugs regressing,
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.

On the eighth rev of software, my dev team dropped on me
Eight devs on sick leave,
Seven bosses angry,
Six specs a-missing,
Five bugs regressing,
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.

On the ninth rev of software, my dev team dropped on me
Nine queries locking,
Eight devs on sick leave,
Seven bosses angry,
Six specs a-missing,
Five bugs regressing,
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.

clip-art-waitingOn the tenth rev of software, my dev team dropped on me
Ten seconds lagging,
Nine queries locking,
Eight devs on sick leave,
Seven bosses angry,
Six specs a-missing,
Five bugs regressing,
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.

On the eleventh rev of software, my dev team dropped on me
Eleven builds a-crashing,
Ten seconds lagging,
Nine queries locking,
Eight devs on sick leave,
Seven bosses angry,
Six specs a-missing,
Five bugs regressing,
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.

On the twelfth rev of software, my dev team dropped on me
Twelve months were wasted,
Eleven builds a-crashing,
Ten seconds lagging,
Nine queries locking,
Eight devs on sick leave,
Seven bosses angry,
Six specs a-missing,
Five bugs regressing,
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release...

Youre-Fired1Final Round...

He sent résumés, to his team gave a whistle,
“This product is dead”, he said this Fo’shizzle.
If you do it at all, then perhaps do it right.
Happy coding to all, and to all a good night!

 

 

 

 

 

A very special thank you to my good friend Maor, who reviewed, QAed and wrote the first verse of the Version of St. Service Bus.

P.S.

This post was supposed to go up on Monday, before Christmas Eve. Unfortunately, I, too, missed my deadline.

Happy New Year.

P.S. 2

My New Year’s Resolution is Retina.

Sunday, January 6, 2013

Patterns of Testable Software

Patterns of Testable SoftwareMy love for test driven development started a few years ago, when I first had a taste of it while working on a product in a startup company. I started out using TypeMock, a really wonderful framework that allows you to isolate the code you want to test from its external dependencies, without having to change the way your code is designed. Within a few months, my team and I completed our work, met our deadline, and we were confident in the product, because we had tests to prove that it works.

Writing Unit Tests is Easy! Good Tests… Less so

Writing unit tests is easy. All you have to do is choose a single use-case of a method that you’re going to test, set up its inputs, and measure the output against some expected result. If it matches, then the test passed. If not, it fails. Rinse and repeat.

Our problems started after that. With the next version, we had to modify our code. No problem, we thought. Our code has unit tests. We can refactor without fear, because our tests will tell us if we’re breaking anything. Except it didn’t. Because of the way our software was written, and because of the way we mocked our dependencies, our tests were tied in to the implementation of the code, rather than the end results. This meant that our tests were brittle, and fragile, because the tests broke whenever we changed our internal implementation. Our tests were starting to “cry wolf”. We abandoned them quickly.

As it turns out, it is not enough “just” to write tests. You actually have to write the production code in a way that is easily testable. Your code has to be written in such a way that you can verify the outcome of a method without depending on internal implementation details. As time passed, I became more and more proficient at doing this, while becoming increasingly frustrated at the difficulties that beginners have. I see many good programmers giving up early, because the tests fail them, because it is difficult to get existing (legacy) code under a good test harness, and because they keep hitting a wall with the question of “how do I test this?”.

Patterns of Testable Software

I have recently begun trying to codify recurring patterns of software that is easily testable: Patterns that developers should strive to conform to, or refactor their code to. Patterns that help developers test the state of a module after running it, as well as to test the behavior of the module when running. Patterns that test the outcome without depending on implementation details.

On Thursday, January 24th, 2013, I am going to present some of these patterns for the first time, at the Toronto ALM User Group. At the time of this writing, there are still some open spots. If you’re in the neighborhood, and are interested in Test Driven Development or writing unit tests in general, I think that you’ll find the material quite rewarding. You may register for free at http://www.meetup.com/Toronto-ALM-User-Group/events/97454932/

I hope to see you there and if I do, I hope you’ll enjoy the session.

Assaf

P.S. Oh, and happy new year!

Tuesday, August 21, 2012

Windows 8 RTM is Available for Evaluation

imageIf you haven’t been living under a rock, then you probably heard of Windows 8. It’s new, it’s different, it moved your cheese to a different time zone.

And it’s freely available for a 90 day evaluation!

You can download the Enterprise edition here.

A few things to note before you try it:

  • This evaluation cannot be upgraded!
  • You will have to install a retail version of Windows when the trial period is over! Don’t say I didn’t warn you…
  • To revert to a previous version of Windows (say, Win-7), you’ll have to do a clean install.

You may wish to try installing it on a virtual machine for evaluation. Personally, I’m partial to Virtual Box, though you can of course create one with any platform.

I’ve already installed it. So far, so good. Everything but my Dell’s fingerprint reader works. Grrrr….

Saturday, June 9, 2012

How Does the LinkedIn Hack Affect Me?

June 10th Update: I added a new GUI tool, to make it easier to check your password’s hash

salted-hash-linkedinIf you have a LinkedIn account, you may be the owner of one of the 6.4 million accounts that have been recently hacked. You should check to be sure. If you have been hacked, your password has been forever compromised in all of your current and future accounts! In this post I will try to help you check whether or not you’re safe, explain what LinkedIn did wrong, and how and why all developers should be more careful.

First Things First: Was I Hacked?

hash-checker-iconI wrote a simple command-line tool that will help you check whether you were hacked or not. What you do is enter your password, and it will check the SHA-1 hash of your password against the published list of hacked hashes, and tell you whether you’re safe or not. Don’t understand what I’m talking about? Never mind. You soon will.

  • Is it safe? Yes. It is an offline tool, and doesn’t connect to the internet in any way – especially not to publish your password, nor is it saved on the disk or added to the list.
  • Don’t believe me? No problem – disconnect your internet connection while using it, then delete the application from your machine, before reconnecting to the internet.
  • Still worried? Still no problem. I’ve published the sources. You can check what I do, and compile the checker yourself.

The hash-checker project is stored at http://hashchecker.codeplex.com. You can get both the binary and the source code there.

The list of compromised hashes can be found here.

Usage:

To check whether you were affected by the aforementioned LinkedIn attack, download the tool and the published list of stolen hashes. Make sure the text file and executable are in the same location.

  1. Run the tool (Icon added to your desktop – you can’t miss it).
  2. To check against the LinkedIn hacked hash list, just type your password and select the hash list file.
  3. Optionally, you may unhide the characters, decide not to pad, or change the number of characters padded.
  4. Press “Search”, and wait a bit. Hopefully, the result will be in green, like mine…

In case you’re wondering about the padding, the list has the first 5 characters zeroed so as not to expose the hashes completely. Pretty much like with credit-card hacks, they leave out the last 4-6 digits.

Here’s a screen shot of my self-testing:

image

I sincerely hope you’re in the clear. And that this is the entire list of compromised hashes…

Either way, take a deep breath, and move on. Time to discuss what happened.

What is a Hash, and What Happened to LinkedIn?

Okay, here’s a crash course of password management. First, in order to authenticate users, most sites store the unique user identifier (username, email address, etc.) in a database, with some form of password.

Plain Text Passwords

The most naïve (and dangerous – never do it like that) solution is to store the password in plain-text, i.e. unencrypted. What these sites do, when you log in, is send the user-name and password (often unencrypted) over the internet, to the site, and then compare the given password with the stored one. If they’re the same, the user authenticated.

The problem is, that if someone manages to hack into the site’s database, that person now has your credentials, and can impersonate you on the site.

That might not seem very harmful, especially if it is not an important site, and you didn’t give it any sensitive information like your Social Security Number, or banking credentials. Unfortunately, that is not true! Statistically speaking, most users reuse their usernames and passwords in multiple sites, including ones where sensitive information is stored!

This means that if you register with www.someunimportantsite.com, with your email address, and password 12345678, it is extremely likely that you used the same password when you registered with the email service, and with www.paypal.com, and many other sites as well.

Knowing this, a malicious hacker may crack someunimportantsite.com, and with the credentials he got there, try to log in and impersonate you on paypal.com! Congratulations – you have now lost money!

Hashes and Hashed Passwords

hero_hash-brownsIn order to protect the passwords, most sites that care about their users (and about malpractice lawsuits), will encrypt their users’ passwords in such a way that it is impossible to derive the password from them. These sites use a form of encryption that is based on a one-way encryption algorithm. Simply put, it is an algorithm that easily and deterministically transforms some plain-text input into an encrypted output, but no algorithm exists that can reverse the operation and derive the input, based on the encrypted output. This encrypted output is called a hash. Two commonly used algorithms are SHA-1 and MD-5. LinkedIn, by the way, use SHA-1.

What these sites do, when you log in, is to encrypt your input, and compare it to the stored encrypted password. If the encrypted input matches the encrypted password, you’ve authenticated.

Problem is, this isn’t good enough.

Rainbow Tables and Rainbow Attacks

imageAs previously mentioned, while it is impossible to reverse a hash, and derive a password, it is easy to generate a hash based on some input. And the same input will always generate the same hash. A hacker will use a rainbow-table, a table that has known passwords and their hashes, to hack a site that stores hashed-passwords.

What the hacker does, is take a hash, look it up in the rainbow table, and if he finds it, he looks up the password that creates it. Congratulations, you’ve now been compromised.

This is what happened with LinkedIn. This is the danger you face.

What Should LinkedIn’s Developers Have Done?

I’ve got no problem with how LinkedIn dealt with the attack, after it happened. My grief is that this fiasco could have been prevented!

What Is a Salted Hash, and How Does it Protect Users?

imageThe problem is that many people reuse passwords, and that many passwords are commonly used by different people: 12345, 1qaz!QAZ, any word in the dictionary, “p@ssw0rd”, and 1337-speak (you pseudo-hacker-cool-boys know who you are). They all appear in every respectable rainbow table.

If a site’s developer would add some random and unique characters to the password, the resulting hash would be vastly different from the hash of the same unsalted password. It would be vastly different from the same password hashed with a different salt.

In short, a salted hash defeats rainbow attacks, because the uniquely salted hash would never appear in it, and no two users would have the same hash, and no user would have the same hash in two different sites.

The great thing is that since the salt cannot be separated from the salted password, the salt doesn’t need to be secret! The site can safely store the salt alongside the salted hash, without fearing for the security.

I repeat – all the site’s developers need to do is create a salt (256 bits of random bytes will suffice) and then hash the password concatenated with the salt:

saltedHash = Algorithm.ComputeHash(password+randomBytes);

Finally, save the salted hash and the salt in the same record in the database.

That is what LinkedIn should have done!

What Can You Do?

Unfortunately, beyond writing your congressman, brow-beating your site’s developers, and boycotting unsafe sites, you can’t do anything about how the passwords are saved. You probably won’t even know how they manage your password until something bad happens and it gets published.

You can, however make sure not to use commonly used passwords, and not make sure that your accounts in sensitive sites have unique, strong passwords, that you don’t reuse.

A password manager tool (I paid for, and love AI RoboForm) can help you keep passwords without having to remember them, and can generate strong random passwords for you.

If this seems like too much work, use levels of security – have one password for unimportant sites, and another safer one for important sites, and a unique one for each extremely sensitive or money-related site (email service, bank, PayPal, etc.).

I hope this helped clear out the air around LinkedIn, hashes, salts, and your own safety.

Assaf.

Tuesday, May 8, 2012

One Team’s First Sprint Planning Session

This post is a bit different than my usual ones. Instead of orating off of my soapbox, I’d like to share a moment of agile bliss that I had today. One of the teams I work with at my customer’s company conducted their first “truly agile” sprint planning session. This post is a recount of the experience.

What Came Before

About a month ago, I began working on agile methodology training for a few teams at my client. The focus of the training was team work-process, mostly using Scrum as a template for how we will do things, with a few Kanban inspired overrides. In my first few sessions, I focused on what a user story is, what a task is, and the concept of focusing the team’s energy on completing user stories as fast as possible.

What I taught them boiled down to a few key take-away points:

  • A user story is an independent increment in functionality that is valuable to a stakeholder, that can be completed by the team in one sprint
  • A task is one developer’s day of work, on one component of the system that is affected by its user story, or work that needs to be done to meet the team’s quality standards.
  • Each team member pulls an unassigned task from the most important, unfinished user story

Initially there were arguments on all points: “Our stories are too big for one sprint”, “What if a task takes more or less than a day?”, “It’s easier to assign a story to a developer”. I had to convince them that these basic ideas are important and valuable, or at least to give them a try.

The Man’s Too Big…

imageWhat we did was take one “too big” user story, and look for some way to break them down in to thinner vertical slices. We ended up with stories that use naïve implementations, or serve a smaller set of clients, or handle a smaller set of scenarios, with further stories that cover the difference in robustness, clients and scenarios.

With regard to the tasks, I suggested that we simply split tasks that feel too big, or join tasks that feel to small, and not to worry about their actual length. We’ll measure the averages and variance after the end of the sprint, and look for better ways to be more predictable as needed. It was more of a let’s simply try it solution.

The Man’s Too Strong!

imageThe longest argument was over whether the first story warranted having multiple developers swarm on the tasks until the story is complete. One developer felt most strongly that it’s a waste of time to have multiple people learn the story, since we’re certain to complete it in time either way. Another developer didn’t want to be pegged in just the server (or client) coding tasks of each story, and felt that owning a whole story gave her more satisfaction. She also expressed concern over the pain of complex merges. Incidentally, it was mostly the QA engineers that immediately accepted the value of working on one story’s tasks in parallel.

Here are the points I made to alleviate their concerns:

  • Having multiple developers learn a story is a great benefit, as it will allow flexibility in task assignment now, and further down the line in future sprints that may have related stories
  • Having more people understand the story will eliminate bottlenecks (and increases the team’s bus factor – a good thing)
  • Regarding the perceived focusing on one component, some other team member chimed in, suggesting that nothing would stop her from taking a client-side task in one story, and a server-related one in the next. I was really happy that other members joined the discussion; it was no longer a teacher-class relationship, but a team of peers
  • I also reminded the team, that since we define a task as the work on one component for a day, and since we focus on completing one story before moving to the next, we are actually reducing the chance of two developers having to manipulate the same object at the same time – which of course reduces the merge conflicts! Further effort can be made to remove the rest of the edge cases almost entirely
  • I further mentioned that there will be at least two members working on each story in any case: one coder, one tester.

Planning the Sprint

imageThe rest went by rather smoothly. The team calculated how many expected work days they have in the sprint. I suggested that we simply add them up, rather than trying to split along the lines of coder / tester days & tasks.

I noticed that the backlog had some free-floating tasks, i.e. not related to any user story. I asked the team about those, stating that unless there is some value to a stakeholder, the task may be a waste of time. The team told me that it was technical debt. After some discussion, we agreed to deduct the tasks from the available work days, rather than creating fake user stories – we have to own up to the fact that we are paying back a debt to the system, not adding value to it.

The team-leader-turned-part-time-product-owner gave a rough stack ranking (there were several items high on the list with the same rank). I suggested that if he doesn’t care which of the same-rank stories we complete first, I’ll set an internal rank by the order that they appeared in the spreadsheet. He accepted.

Next we started explaining and breaking the stories. Traditionally, the PO explains the stories in the planning session’s first half, and the team breaks them down in the second. Since the PO in this case is a team member, and fully savvy of the product and solution, I suggested that we simply run down the list until we have as many tasks as we have work days (see how that works out?).

The first story was easy. We created 4 coding tasks and 2 quality related tasks.

Break it Down!

imageThe second story was a whopper. We knew it would take most of the sprint. One developer suggested that we split the story by data source – one part retrieving one source’s data, the other part retrieving the second source. the team leader shook his head. “The customer needs both”, he said. There is no value in supplying just one. We’ll split it some other way, if we have to”.

You couldn’t erase the smile from my face with truckload of erasers. I knew for sure that he got the idea of what a user story is!

With the second, larger, story, the team leader frowned and looked at the task list. “This doesn’t feel right” he said. When asked what the problem was, he said that one task is too big – it will take longer than a day. The team then split it in two parts, and added a third task.

And we were done estimating and committing. Just like that.

All in all, we spent about an hour of planning, 2-3 minutes of which were spent “estimating”, which is to say deciding how to split that “too big” task.

An Agile Manager in a Brave New World

At the end of the meeting, the team leader expressed his regrets that we don’t have a full-time technical-product-manager (TPM) which is the closest role they have to a Product Owner. The group manager, who was present for the Sprint Planning picked up the impediment and answered “Let’s take it offline; I’ll try to get one to join the team ASAP”.

Yes sir! The manager became the Scrum Master’s Scrum Master, as per my suggestion in my previous post.

Here’s What We Didn’t Do

There are several things the team didn’t do:

  • Defining proper user stories – I didn’t bother following the ceremony of form with the stories, declaring the reasoning and the interested party. I ignored this because I wanted to avoid too much change-shock at once. Next sprint we’ll focus on that. For now, I felt that understanding the concept of value was more – well – valuable
  • Estimate the effort involved in the stories. Personally? I hate estimations. I think they’re useless, most of the time. I noticed that most of the time spent in a sprint planning session is wasted on planning poker, story points and ideal hours, and in the end – they don’t help the project’s predictability in any noticeable way. Instead, we’ll measure, and seek a low-variance way to split the tasks. That will be better. and cheaper

In the End

All in all, it was a great first “agile day” for the team. It was fast, to the point, and everybody participated in every part of the session. Tomorrow I will work with them on building their Scrum-board, and walk them through their first “real” daily meeting. I can’t wait!

So, what do you think? Want to hear more such experiences? Is there anything I did that you like? Something I could have done better? Please drop a comment. Don’t be a stranger!

P.S. A special thank you goes to Mark Knopfler, lead vocals and guitar legend from the Dire Straits, for inspiring the titles of two sections, in his wonderful song.