Change Happens
Change is a Good Thing

So What is the Problem?

Change is Changing!

The Pros
The Cons
Changing is Difficult. Not Changing is Fatal

The Way of Tomorrow Today

“Your software doesn’t work!” states the angry voice on the other end.
“Uhhh… Could you be more explicit, sir?” asks the still bewildered techy, trying to figure out what’s going on.
“Your software doesn’t #&@<ing work!”
No, we don’t think that you’re making it up, that you’re dense, or that you’re bothering us with trivialities. We just want to know what kind of a bug you might be dealing with now:
Especially if you are a developer or tester, you might have a special configuration file with specific setting that you are using. Please provide any non-sensitive configuration, or at least describe it. Anything you know about how you use the software can shorten the time it takes to fix the problem.
Sometimes a bug will consistently appear whenever the application is executed. In other cases it might be limited to happening only sporadically:
We’re all in this together – developers, testers, operations – we usually (should) have the same interest – to make sure that we create a better product. So please – pretty please – with relish(!): be professional.
Thanks in advance,
Assaf – a developer.
’Twas the week O’releasing, and all through the teams,
Not a programmer testing, at least so it seems.
The builds barely holding, the crashes abound,
In hopes that some luck, or a safe place be found...
[Enter the choir. You can sing along. You know you want to…]
On the first rev of software, my dev team dropped on me
A piece of software that I can't release.
On the second rev of software, my dev team dropped on me
Two breaking smoke tests,
And a piece of software that I can't release.
On the third rev of software, my dev team dropped on me
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.
On the fourth rev of software, my dev team dropped on me
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.
On the fifth rev of software, my dev team dropped on me
Five bugs regressing,
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.
On the sixth rev of software, my dev team dropped on me
Six missing specs,
Five bugs regressing,
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.
On the seventh rev of software, my dev team dropped on me
Seven bosses angry,
Six missing specs,
Five bugs regressing,
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.
On the eighth rev of software, my dev team dropped on me
Eight devs on sick leave,
Seven bosses angry,
Six specs a-missing,
Five bugs regressing,
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.
On the ninth rev of software, my dev team dropped on me
Nine queries locking,
Eight devs on sick leave,
Seven bosses angry,
Six specs a-missing,
Five bugs regressing,
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.
On the tenth rev of software, my dev team dropped on me
Ten seconds lagging,
Nine queries locking,
Eight devs on sick leave,
Seven bosses angry,
Six specs a-missing,
Five bugs regressing,
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.
On the eleventh rev of software, my dev team dropped on me
Eleven builds a-crashing,
Ten seconds lagging,
Nine queries locking,
Eight devs on sick leave,
Seven bosses angry,
Six specs a-missing,
Five bugs regressing,
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release.
On the twelfth rev of software, my dev team dropped on me
Twelve months were wasted,
Eleven builds a-crashing,
Ten seconds lagging,
Nine queries locking,
Eight devs on sick leave,
Seven bosses angry,
Six specs a-missing,
Five bugs regressing,
Four tests a-failing,
Three days till deadline,
Two breaking smoke tests,
And a piece of software that I can't release...
He sent résumés, to his team gave a whistle,
“This product is dead”, he said this Fo’shizzle.
If you do it at all, then perhaps do it right.
Happy coding to all, and to all a good night!
A very special thank you to my good friend Maor, who reviewed, QAed and wrote the first verse of the Version of St. Service Bus.
This post was supposed to go up on Monday, before Christmas Eve. Unfortunately, I, too, missed my deadline.
Happy New Year.
My New Year’s Resolution is Retina.
My love for test driven development started a few years ago, when I first had a taste of it while working on a product in a startup company. I started out using TypeMock, a really wonderful framework that allows you to isolate the code you want to test from its external dependencies, without having to change the way your code is designed. Within a few months, my team and I completed our work, met our deadline, and we were confident in the product, because we had tests to prove that it works.
Writing unit tests is easy. All you have to do is choose a single use-case of a method that you’re going to test, set up its inputs, and measure the output against some expected result. If it matches, then the test passed. If not, it fails. Rinse and repeat.
Our problems started after that. With the next version, we had to modify our code. No problem, we thought. Our code has unit tests. We can refactor without fear, because our tests will tell us if we’re breaking anything. Except it didn’t. Because of the way our software was written, and because of the way we mocked our dependencies, our tests were tied in to the implementation of the code, rather than the end results. This meant that our tests were brittle, and fragile, because the tests broke whenever we changed our internal implementation. Our tests were starting to “cry wolf”. We abandoned them quickly.
As it turns out, it is not enough “just” to write tests. You actually have to write the production code in a way that is easily testable. Your code has to be written in such a way that you can verify the outcome of a method without depending on internal implementation details. As time passed, I became more and more proficient at doing this, while becoming increasingly frustrated at the difficulties that beginners have. I see many good programmers giving up early, because the tests fail them, because it is difficult to get existing (legacy) code under a good test harness, and because they keep hitting a wall with the question of “how do I test this?”.
I have recently begun trying to codify recurring patterns of software that is easily testable: Patterns that developers should strive to conform to, or refactor their code to. Patterns that help developers test the state of a module after running it, as well as to test the behavior of the module when running. Patterns that test the outcome without depending on implementation details.
On Thursday, January 24th, 2013, I am going to present some of these patterns for the first time, at the Toronto ALM User Group. At the time of this writing, there are still some open spots. If you’re in the neighborhood, and are interested in Test Driven Development or writing unit tests in general, I think that you’ll find the material quite rewarding. You may register for free at http://www.meetup.com/Toronto-ALM-User-Group/events/97454932/
I hope to see you there and if I do, I hope you’ll enjoy the session.
Assaf
P.S. Oh, and happy new year!
If you haven’t been living under a rock, then you probably heard of Windows 8. It’s new, it’s different, it moved your cheese to a different time zone.
And it’s freely available for a 90 day evaluation!
You can download the Enterprise edition here.
A few things to note before you try it:
You may wish to try installing it on a virtual machine for evaluation. Personally, I’m partial to Virtual Box, though you can of course create one with any platform.
I’ve already installed it. So far, so good. Everything but my Dell’s fingerprint reader works. Grrrr….
If you have a LinkedIn account, you may be the owner of one of the 6.4 million accounts that have been recently hacked. You should check to be sure. If you have been hacked, your password has been forever compromised in all of your current and future accounts! In this post I will try to help you check whether or not you’re safe, explain what LinkedIn did wrong, and how and why all developers should be more careful.
I wrote a simple command-line tool that will help you check whether you were hacked or not. What you do is enter your password, and it will check the SHA-1 hash of your password against the published list of hacked hashes, and tell you whether you’re safe or not. Don’t understand what I’m talking about? Never mind. You soon will.
The hash-checker project is stored at http://hashchecker.codeplex.com. You can get both the binary and the source code there.
The list of compromised hashes can be found here.
To check whether you were affected by the aforementioned LinkedIn attack, download the tool and the published list of stolen hashes. Make sure the text file and executable are in the same location.
In case you’re wondering about the padding, the list has the first 5 characters zeroed so as not to expose the hashes completely. Pretty much like with credit-card hacks, they leave out the last 4-6 digits.
Here’s a screen shot of my self-testing:
I sincerely hope you’re in the clear. And that this is the entire list of compromised hashes…
Either way, take a deep breath, and move on. Time to discuss what happened.
Okay, here’s a crash course of password management. First, in order to authenticate users, most sites store the unique user identifier (username, email address, etc.) in a database, with some form of password.
The most naïve (and dangerous – never do it like that) solution is to store the password in plain-text, i.e. unencrypted. What these sites do, when you log in, is send the user-name and password (often unencrypted) over the internet, to the site, and then compare the given password with the stored one. If they’re the same, the user authenticated.
The problem is, that if someone manages to hack into the site’s database, that person now has your credentials, and can impersonate you on the site.
That might not seem very harmful, especially if it is not an important site, and you didn’t give it any sensitive information like your Social Security Number, or banking credentials. Unfortunately, that is not true! Statistically speaking, most users reuse their usernames and passwords in multiple sites, including ones where sensitive information is stored!
This means that if you register with www.someunimportantsite.com, with your email address, and password 12345678, it is extremely likely that you used the same password when you registered with the email service, and with www.paypal.com, and many other sites as well.
Knowing this, a malicious hacker may crack someunimportantsite.com, and with the credentials he got there, try to log in and impersonate you on paypal.com! Congratulations – you have now lost money!
In order to protect the passwords, most sites that care about their users (and about malpractice lawsuits), will encrypt their users’ passwords in such a way that it is impossible to derive the password from them. These sites use a form of encryption that is based on a one-way encryption algorithm. Simply put, it is an algorithm that easily and deterministically transforms some plain-text input into an encrypted output, but no algorithm exists that can reverse the operation and derive the input, based on the encrypted output. This encrypted output is called a hash. Two commonly used algorithms are SHA-1 and MD-5. LinkedIn, by the way, use SHA-1.
What these sites do, when you log in, is to encrypt your input, and compare it to the stored encrypted password. If the encrypted input matches the encrypted password, you’ve authenticated.
Problem is, this isn’t good enough.
As previously mentioned, while it is impossible to reverse a hash, and derive a password, it is easy to generate a hash based on some input. And the same input will always generate the same hash. A hacker will use a rainbow-table, a table that has known passwords and their hashes, to hack a site that stores hashed-passwords.
What the hacker does, is take a hash, look it up in the rainbow table, and if he finds it, he looks up the password that creates it. Congratulations, you’ve now been compromised.
This is what happened with LinkedIn. This is the danger you face.
I’ve got no problem with how LinkedIn dealt with the attack, after it happened. My grief is that this fiasco could have been prevented!
The problem is that many people reuse passwords, and that many passwords are commonly used by different people: 12345, 1qaz!QAZ, any word in the dictionary, “p@ssw0rd”, and 1337-speak (you pseudo-hacker-cool-boys know who you are). They all appear in every respectable rainbow table.
If a site’s developer would add some random and unique characters to the password, the resulting hash would be vastly different from the hash of the same unsalted password. It would be vastly different from the same password hashed with a different salt.
In short, a salted hash defeats rainbow attacks, because the uniquely salted hash would never appear in it, and no two users would have the same hash, and no user would have the same hash in two different sites.
The great thing is that since the salt cannot be separated from the salted password, the salt doesn’t need to be secret! The site can safely store the salt alongside the salted hash, without fearing for the security.
I repeat – all the site’s developers need to do is create a salt (256 bits of random bytes will suffice) and then hash the password concatenated with the salt:
saltedHash = Algorithm.ComputeHash(password+randomBytes);
Finally, save the salted hash and the salt in the same record in the database.
That is what LinkedIn should have done!
Unfortunately, beyond writing your congressman, brow-beating your site’s developers, and boycotting unsafe sites, you can’t do anything about how the passwords are saved. You probably won’t even know how they manage your password until something bad happens and it gets published.
You can, however make sure not to use commonly used passwords, and not make sure that your accounts in sensitive sites have unique, strong passwords, that you don’t reuse.
A password manager tool (I paid for, and love AI RoboForm) can help you keep passwords without having to remember them, and can generate strong random passwords for you.
If this seems like too much work, use levels of security – have one password for unimportant sites, and another safer one for important sites, and a unique one for each extremely sensitive or money-related site (email service, bank, PayPal, etc.).
I hope this helped clear out the air around LinkedIn, hashes, salts, and your own safety.
Assaf.
This post is a bit different than my usual ones. Instead of orating off of my soapbox, I’d like to share a moment of agile bliss that I had today. One of the teams I work with at my customer’s company conducted their first “truly agile” sprint planning session. This post is a recount of the experience.
About a month ago, I began working on agile methodology training for a few teams at my client. The focus of the training was team work-process, mostly using Scrum as a template for how we will do things, with a few Kanban inspired overrides. In my first few sessions, I focused on what a user story is, what a task is, and the concept of focusing the team’s energy on completing user stories as fast as possible.
What I taught them boiled down to a few key take-away points:
Initially there were arguments on all points: “Our stories are too big for one sprint”, “What if a task takes more or less than a day?”, “It’s easier to assign a story to a developer”. I had to convince them that these basic ideas are important and valuable, or at least to give them a try.
What we did was take one “too big” user story, and look for some way to break them down in to thinner vertical slices. We ended up with stories that use naïve implementations, or serve a smaller set of clients, or handle a smaller set of scenarios, with further stories that cover the difference in robustness, clients and scenarios.
With regard to the tasks, I suggested that we simply split tasks that feel too big, or join tasks that feel to small, and not to worry about their actual length. We’ll measure the averages and variance after the end of the sprint, and look for better ways to be more predictable as needed. It was more of a let’s simply try it solution.
The longest argument was over whether the first story warranted having multiple developers swarm on the tasks until the story is complete. One developer felt most strongly that it’s a waste of time to have multiple people learn the story, since we’re certain to complete it in time either way. Another developer didn’t want to be pegged in just the server (or client) coding tasks of each story, and felt that owning a whole story gave her more satisfaction. She also expressed concern over the pain of complex merges. Incidentally, it was mostly the QA engineers that immediately accepted the value of working on one story’s tasks in parallel.
Here are the points I made to alleviate their concerns:
The rest went by rather smoothly. The team calculated how many expected work days they have in the sprint. I suggested that we simply add them up, rather than trying to split along the lines of coder / tester days & tasks.
I noticed that the backlog had some free-floating tasks, i.e. not related to any user story. I asked the team about those, stating that unless there is some value to a stakeholder, the task may be a waste of time. The team told me that it was technical debt. After some discussion, we agreed to deduct the tasks from the available work days, rather than creating fake user stories – we have to own up to the fact that we are paying back a debt to the system, not adding value to it.
The team-leader-turned-part-time-product-owner gave a rough stack ranking (there were several items high on the list with the same rank). I suggested that if he doesn’t care which of the same-rank stories we complete first, I’ll set an internal rank by the order that they appeared in the spreadsheet. He accepted.
Next we started explaining and breaking the stories. Traditionally, the PO explains the stories in the planning session’s first half, and the team breaks them down in the second. Since the PO in this case is a team member, and fully savvy of the product and solution, I suggested that we simply run down the list until we have as many tasks as we have work days (see how that works out?).
The first story was easy. We created 4 coding tasks and 2 quality related tasks.
The second story was a whopper. We knew it would take most of the sprint. One developer suggested that we split the story by data source – one part retrieving one source’s data, the other part retrieving the second source. the team leader shook his head. “The customer needs both”, he said. There is no value in supplying just one. We’ll split it some other way, if we have to”.
You couldn’t erase the smile from my face with truckload of erasers. I knew for sure that he got the idea of what a user story is!
With the second, larger, story, the team leader frowned and looked at the task list. “This doesn’t feel right” he said. When asked what the problem was, he said that one task is too big – it will take longer than a day. The team then split it in two parts, and added a third task.
And we were done estimating and committing. Just like that.
All in all, we spent about an hour of planning, 2-3 minutes of which were spent “estimating”, which is to say deciding how to split that “too big” task.
At the end of the meeting, the team leader expressed his regrets that we don’t have a full-time technical-product-manager (TPM) which is the closest role they have to a Product Owner. The group manager, who was present for the Sprint Planning picked up the impediment and answered “Let’s take it offline; I’ll try to get one to join the team ASAP”.
Yes sir! The manager became the Scrum Master’s Scrum Master, as per my suggestion in my previous post.
There are several things the team didn’t do:
All in all, it was a great first “agile day” for the team. It was fast, to the point, and everybody participated in every part of the session. Tomorrow I will work with them on building their Scrum-board, and walk them through their first “real” daily meeting. I can’t wait!
So, what do you think? Want to hear more such experiences? Is there anything I did that you like? Something I could have done better? Please drop a comment. Don’t be a stranger!
P.S. A special thank you goes to Mark Knopfler, lead vocals and guitar legend from the Dire Straits, for inspiring the titles of two sections, in his wonderful song.