A Programmer’s Journey

1: Your Role in Software Quality

Once there was a bright young programmer named Jon. Jon loved designing and building software, but something was nagging at him. As he worked on larger projects with more complexity, bugs seemed to quickly get out of hand, and neither he nor his teammates were able to regain control. In one project, Jon reflected back on all the nights and weekends he had worked trying to conquer a  mountain of bugs. “All these extra hours haven’t really changed anything,” he sadly concluded. “Getting quality into software must be more fundamental than the number of hours you work.”

One day Jon joined a new project whose team really seemed to have its act together. Jon decided he would become a student of programmer quality and learn from his new teammates.

Jon’s first visit was to Ruth, the programmer with the lowest bug count of anyone on the team. “What’s your secret?” asked Jon. “Why do you have fewer bugs than anyone else?” Ruth smiled thoughtfully, then looked out the window and gestured to a line of trees in rocky ground that had been turned up and displaced by a recent storm. “Do you see those trees over there? Their roots were not deep, so they were easily uprooted.” Then she pointed to the other side of the street, where a line of trees stood tall. “But those trees over there held their ground. Their deep roots gave them integrity. That’s how I am about the software I write: I’m a quality gate. That means I don’t release anything for formal test until I’ve tested it thoroughly and have fixed all the obvious bugs.”

Jon considered for a long moment, then asked, “But aren’t you under pressure to deliver software in a certain time period? What happens when the end of sprint comes and you haven’t finished testing and debugging?” Again Ruth smiled. “Do you want to be steamrollered by the quality process or do you want to be an active participant in it?” “An active participant!” said Jon without hesitation. “Well then,” continued Ruth, “that requires integrity. Like the trees, there are going to be storms: pressure to deliver the software before it’s ready. But if you’re a quality gate, your job is to hold firm—that’s what gives the quality process integrity.

“I’ll admit it was a little scary holding firm at first, but my manager and team have come to realize that getting software sooner rather than later is of no use if it’s buggy,” Ruth added. “And don’t get the idea that I’m constantly late on sprint work. I’ve improved my estimating. I set realistic expectations for what I can accomplish in a sprint that includes plenty of time for testing and iteration.”

“I can see the value of what you’re saying,” said Jon, “but isn’t quality QA’s job? With all that extra testing it seems like you’re doing QA’s job as well as your own.” Ruth laughed. “Our QA team is great, but quality is my responsibility—and no one else’s! Since I create the software, I’m the only one who can affect its quality.”

2: Building in Quality

Jon’s next visit was to Craig, a middle-aged developer who seemed to have everyone’s respect for solid code. On his desk was a biography of Michelangelo. “I’ve been looking at some of the code you’ve written in source control, and I must say, it’s absolutely pristine.” admitted Jon. “Your code is so clean it screams quality. Can you tell me why your code looks so much better than anyone else’s?”

Craig blushed but eagerly drew up a chair. “A few years ago, I had to extend some code that was several years old. It wasn’t written very well: it was overly complex, hard to follow, and had few comments. I couldn’t understand how much of it worked, which made it a real nightmare for me. Many of my changes backfired. The worst thing was, I was the author of this code! Enough time had gone by that I couldn’t make heads or tails of it.” Jon’s eyes widened in appreciation. Craig continued, “I decided then and there that I would only write quality code from that point on. Code has to be understandable and maintainable, or it’s not worth writing.

“When I write code now, the design must be solid, if not elegant. The code must be cleanly structured without complexity. Oh, it doesn’t always start out like that; there are plenty of missteps and rework along the way. But that’s how I end up. It takes dedication to good software engineering principles—like Separation of Concerns, DRY, and Defensive Programming; and sometimes a lot of refactoring. The code also needs to be proven by passing tests. Only then am I finished.”

“Hmm…” mused Jon. “I have to tell you, I think I am already doing most of those things. Why doesn’t my code look like yours? I try to avoid complexity, I refactor…” “You’re doing those things, but are you doing them enough?” asked Craig. “What do you mean?” asked a puzzled Jon.

“What I’m asking,” said Craig, “is how fully are you applying those software engineering principles? For example, let’s take Don’t Repeat Yourself. When our team first started discussing DRY a few years back, most of our developers said they were already following this principle and had a good set of classes in place. What we found, though, was that we had barely scratched the surface. We found a code clone analysis tool for our development environment. When we ran it, we discovered we had over 100,000 lines of duplicate code in our project! When we started thinking about all the ways we could apply DRY, new opportunities opened up all over the place. Our front-end developers replaced our CSS files with LESS files, which allowed us to centralize things like colors in variables rather than repeating them throughout many style rules. Our database programmer started using views to avoid duplicate select statements and joins in stored procedures.

“I’m also asking whether you are finishing the job,” Craig continued. “I doubt my first draft of code looks much better than yours, but I refine it and iterate until I am satisfied with it. I guess that’s the craftsman in me. I like to picture myself as the sculptor removing all the unnecessary bits from a slab of marble to reveal the masterpiece within. Did you know Michelangelo spent three years on the statue of David and four years on the Sistine Chapel?” Jon admitted he didn’t. “It’s when your code begins working that you may be tempted to say you’re finished—but you mustn’t stop before your code is clean and complete and proven.”

“Allow me to recap,” Jon offered. “You need to use good software engineering principles and apply them fully, but you also need to be a craftsman. You can do all the right things, but if you don’t iterate to the finish line you’re really only half-done.” “Now you’ve got it!” replied Craig with satisfaction.

3: Testing Your Own Work

Jon next met up with Jai, one of the testers in the QA group. Jon asked, “Our team really seems to know what it is doing when it comes to quality. I’ve talked to some of my fellow developers, but I wanted to get your opinion on why that is?” Jai thought for a moment, then responded, “It is a lot better now but it wasn’t always that way. I’d like to take credit for it, but to be honest I think much of the shift in quality has come from the developers: they have really committed to being part of the quality process instead of just reacting to it.”

“And how has that made things different for you?” asked Jon. “Well,” said Jai, “It used to be very easy to find bugs in the solution. When we would get a new build, we would find dozens of bugs our very first day with it. But that has all changed. The developers find the obvious bugs and fix them before they release software to us. As a result, we have to work much harder to find issues.”

“That almost sounds like it has made a problem for you.” said Jon. “Maybe management will conclude the developers are doing such as great job on quality that the QA group isn’t necessary?” Jai smiled. “Oh, there’s no danger of that.” He pointed to a poster on the wall that read, “’If you do your job, I can do mine.’” Jai continued, “We are now able to focus on what we should have been doing all along: exhaustive testing, finding the subtle issues, and making sure we have good coverage with our automated tests.

“We do some kinds of testing the developers don’t do, such as UI automation tests, performance tests, and security audits. It was hard to get all those things done previously because we had our hands full just cataloging all the bugs we found in each build. Now that the developers also build automated tests, this gives us a great starting point for QA’s own tests. The software has also become easier to write tests for as a result.”

4: Functional Testing

Jon next wanted to spend some time with the programmer who knew the most about testing their own work; but when he inquired who that would be, he was given two names: Brian and Howie. He went to visit Brian first, only to find Howie was there too, working right alongside him. “Can one of you spare some time to discuss developer testing with me?” asked Jon. “You should talk to both of us,” said Brian. “We work as a pair, and we help test each other’s work.”

“Great!” said Jon appreciatively. “I presume you’re writing unit tests.” said Jon. “Every place I’ve ever worked, the developers have been expected to write unit tests.” “We do write unit tests whenever we create or update a class,” said Brian. “And integration tests whenever we create a server action or API.” added Howie. “But we’ve just finished implementing a story, and we need to test that.”

“Test it how?” asked Jon. “There are so many kinds of testing to choose from. There’s unit testing, integration testing, system testing, black box testing, white box testing, UI testing… the list is endless. Which tests should a developer be doing when they finish a story?”

“That’s easy—Functional Testing.” said Brian. “You mean, you only do one kind of testing?” asked Jon, a bit surprised. Brian continued, “Functional testing has to come first. There’s no point in doing any other kind of testing until you know your code works when used properly. Otherwise what you’ve done is wrong or incomplete.” Howie added, “Once your functional tests pass, you’re ready for other kinds of testing such as hostile testing.”

“All right,” said Jon as he took careful notes. “Let’s discuss Functional Testing then.” Howie immediately jumped to his feet, donned a headset, and announced in a loud voice, “My name is Arthur, how may I be of service?” Jon paused, not understanding. “What Howie means,” interjected Brian, “is that we’re testing the call center module today, and we have a persona for a call center rep named Arthur. Howie likes to do a little method acting when we’re doing functional testing. He even dresses the part sometimes.”

Jon said, “I thought personas were just for the UX group.” “No, they’re for everyone.” corrected Brian. Each member of the team must understand our different users and what they need to accomplish.” “’As a call center rep, I need to easily call up customer information during phone calls.’” said Howie, quoting the first part of a story. Brian added, “We have conditions of acceptance for what the software needs to do for a call center rep. As we test our module, we’ll go through the same sequences a call center rep would, looking to confirm each condition of acceptance along the way.”

Jon observed as the two set up initial conditions for their test and walked through a sequence they had scripted. After each user interaction, they checked that certain conditions were true. “Where did you get that script?” asked Jon. “We created it from the story and its conditions of acceptance.” said Brian. “I imagine you also used your knowledge of the code to design the test.” said Jon. “You couldn’t be more wrong.” said Howie. “In Functional Testing, we only use the requirements. We have to put aside what we know of the internals.”

“And teaming up as a pair? How does that work?” asked Jon. “We programmers have blind spots when it comes to testing our own work.” said Brian. “Having a second set of eyes helps find more of the issues.”

Jon watched the pair complete their test sequence. Two issues were found, which were quickly addressed. In the second run of the test sequence, everything seemed to function as expected, so he asked “It looks like you’re done here. That didn’t take long.” “We’re far from finished.” said Brian. “We need to test variations in the sequence to exercise different options the user can select. We need to test the call center rep and call center supervisor roles.” “And,” said Howie, “We need to test on several different browsers and devices.”

“One last question,” said Jon. “How do you know when you’ve done enough testing?” “At a minimum, we’re expected to find all the obvious problems.” said Howie. “If we didn’t do that,” added Brian, “All that would happen is that QA would rapidly find issues and run up a large bug list.” Jon nodded thoughtfully: he’d been in exactly that place in his last project.

5: Hostile Testing

Jon was curious what developers should do after completing Functional Testing. He visited Ross, an energetic young man with a reputation for thorough testing. He knew he was in for an interesting meeting when he noticed a plaque on the wall that read “Don’t Be Evil.” Jon asked, “Can you tell me about hostile testing?” “I can do better than that—I’ll show you.” said Ross.

“Today I’m going to be testing a product ordering sequence—hostilely,” said Ross. “In fact, you can help. Why don’t you sit down over here and begin an order for an X100 exercise machine.” Jon complied, and soon had an order in progress. He noted the X100 he was ordering was the last one in inventory. “Okay, now what?” asked Jon. “Well, what kind of hostile tests can we perform?” asked Ross. Jon considered, and offered a number of unexpected interactions he could make on the order screen such as leaving out information; entering invalid data in some fields; trying to add the same product multiple times; and closing the browser window mid-order and later returning to the site. “Those are all good,” said Ross, “but I’ve already tested everything I can think of in a single-user interaction. It’s time to really challenge the software. We need to be evil.

“We’re going to return to this exact starting point a number of times,” said Ross, “Each time we’ll complicate the order-in-progress with other interactions in parallel.” Jon watched as Ross started another order for the same SKU, to confirm the software would not permit double-ordering of the same item. In the next test, an administrator attempted to delete the X100 product while an order was in progress. In another test the user’s account was put on hold by an administrator mid-way through the ordering process. In yet another test the customer’s shipping address was changed while an order was in progress.

“Come on,” coaxed Ross, “Give me some ideas I haven’t thought of.” Jon thought furiously, then came up with “Let’s return to the site to complete the order, but in two browser sessions at the same time,” and “We could continue the order in one browser session and cancel it in another.” “Excellent! You’re learning!” Ross said with approval.

Jon left with a newfound-appreciation for ways to attack software. In a way he felt dirty, but he knew the application would be incredibly solid once it was able to stand up to all of these abuses.

6: Automated Testing

Jon next met with Alice, a developer on the team who was also the build master. “How important is automated testing around here?” asked Jon. “Well, it’s absolutely essential.” said Alice. “I don’t think we’d have a fraction of the confidence we do in our quality without it.”

“What kind of automated tests does our team have?” asked Jon. “Ideally, we want unit tests for all of our classes, integration tests for our services and APIs, and UI tests for end-to-end sequences.” said Alice. “Which of those do you have the most of?” asked Jon. Turning to a nearby whiteboard, Alice sketched a triangle on the white board divided into 3 sections: UI at the top, Services in the middle, and Unit at the bottom. “This is our automation pyramid. The largest number of tests are unit tests because there are many small units of the software to test such as classes. The next largest group is the Services layer, which are integration tests of server actions, services, and APIs. The smallest number of tests are the UI tests, which cover end-to-end sequences. The percentage of tests we have out of the full set we’d like to have is what we call test coverage; right now our test coverage is about 80%.”

“And who writes all these tests?” asked Jon, afraid he knew the answer. “Unit tests and integration tests are created by us, the developers.” said Alice. “QA creates the UI automation tests. QA will sometimes take one of our integration tests or unit tests and extend them.” “It seems like a lot of work,” said Jon. “It is a lot of work,” admitted Alice, “But it’s far less work than manual testing, which has to be repeated over and over whenever there’s a new build.”

Jon volunteered, “On my last project we were supposed to write unit tests, but we always ran out of time.” Alice responded, “That’s probably because you did them last. We’ve found we get more of our tests written if we use a test-first approach. That means we write the test before we write the implementation, instead of the other way around.” Jon pondered what it would be like to write tests before the code to be tested even existed, a new concept for him.

Alice continued, “Of course, all of these tests wouldn’t amount to much unless we run them whenever the software changes. That’s the beauty of our Continuous Integration system: tests get run every time the software is modified.” Jon perked up. “Actually, I’ve been meaning to ask you about that,” he said. “One thing I don’t like around here is how long it takes to make a check-in; I’ve waited as long as 20 minutes at times for a check-in, and if a check-in is rejected that gets multiplied two or three times over by the time I’ve corrected the issue and made the build system happy.”

“And what,” asked Alice pointedly, “would the result have been if the CI system didn’t reject your check-in?” Jon thought for a moment, then admitted “There would have been breaking changes that affected the rest of the team. I begin to understand. The CI system is protecting us from each other.”

“Let me tell you what things were like before we put CI in place,” said Alice. “We have 10 developers working in 3 time zones. In any 24-hour period, there’s the possibility of someone damaging the software because they made a mistake. For example, a UX developer would add a CSS file to the solution, but forget to add the new file to source control; or a back-end developer would change a class, unaware that the change impacted other parts of the solution. This broke things for the other team members and was highly disruptive.

“Once we realized how precarious a situation we were in, we decided to invest in Continuous Integration; and we all wish we had done it sooner. Now—as you’ve discovered—a breaking change is not permitted to pollute the solution. It’s a safeguard that protects us from each other. Once a feature is complete and working, we make sure we have good coverage in our check-in tests. With that safety net in place, it’s impossible to check in a change that breaks a feature. As long we keep this up our team moves forward, not backward.”

7: Analyzing and Debugging

Jon returned to his work area, his mind full of new concepts he was eager to put into practice. Right now, though, he needed to fix some bugs. He returned his attention to a problem that had been frustrating him since yesterday. Just then, lead developer Jeff dropped by. “How’s it going? Anything I can help with?” asked Jeff. “Actually,” said Jon, “Maybe you can give me some advice. I’m not making any headway on this problem in my JavaScript code. Sometimes it works and sometimes it doesn’t.”

“The best thing to do with a problem like that,” said Jeff, “Is explain it to somebody else. Why don’t you walk me through the code and explain what isn’t working?” Jon wasn’t sure that would help, but was willing to try. “Well, it’s silly really—the code isn’t all that long. Here’s my function to compute the total tax for an order. There’s a loop that cycles through each item in the shopping cart, and inside the loop we perform an Ajax call to look up the tax for each item. By the time the total tax is displayed on the page, sometimes it’s right but often it isn’t.”

Jeff took a moment to study the code, then asked “I see you’re storing the tax that is returned.” “That’s right,” said Jon. “The success function for the Ajax call stores the tax in the array element for the current item—oh wait, I see what’s wrong now. The loop variable has surely changed by then, so I can’t use it to index the item array. I don’t know why I didn’t see it earlier.”

“Explaining problems to others really helps you see things.” said Jeff. “What can you do to fix this?” “Well,” said Jon, “I think I can just move this Ajax code into a function and pass the loop variable to the function. That should keep the value unchanged while the function executes.” “I agree,” said Jeff. “That’s just what I’d do.”

8: Fixing Bugs Completely

Jon thanked Jeff for assisting him in debugging his JavaScript problem. “Thanks for your help. I’ll check this fix in and get going on my next bug,” he said. “Not so fast!” said Jeff. “You’ve fixed your bug, but you haven’t fixed it completely.” Jon was caught off guard. “What do you mean, ‘completely’? I’ve fixed the issue, and I’ve tested the fix. What else is there?”

“Are you sure this code change doesn’t have any side effects? It’s of no use to fix a problem if it creates another one.” Jon spent some time looking at the code and where it was used. Finally he announced “Okay. I’m sure there’s no negative impact from this change. Am I done now?”

“Not yet you’re not,” said Jeff. “Does this problem exist anywhere else in the code?” asked Jeff. Jon considered. “Well, I don’t know. It’s possible I’ve made the same mistake in other places.” “You need to find them, fix them, and test them,” said Jeff. “Go ahead—I’ll wait around.” Chagrined, Jon searched for other places where he might have made the same mistake. He found two other instances and corrected the code, then tested the fixes. “Okay, said Jon. Surely now I’m finished.”

“One more thing,” said Jeff. “This particular bug needs to never happen again. Ever. What changes are you going to put in place so that there is never a recurrence?” Jon considered, then said “I thought I was pretty good with Ajax, but I never thought about Ajax calls in the context of a loop before. I won’t make this error again, and I’ve already fixed the code. What else is there?” “I think it would be a good idea to bolster your understanding of asynchronous patterns in Ajax,” said Jeff. “I’ll send you a link to some good content about that. In addition, I’d like you to share this bug and what you learned from it at our weekly developer discussion next week. The other developers could benefit from being reminded about correct asynchronous patterns.”

9: Bug Causes and Remedies

Jon’s next meeting was the weekly developer meeting. “Several of you are having trouble resolving issues,” said the development manager, “so I’d like us to discuss them as a group. Who’d like to go first?”

Brian volunteered, and put some code up on the screen. “I’ve been struggling to get this database update code working right all week,” he said. Several developers asked questions as they pored over the code, then Ruth spoke up “I think you need to update your tables in a different order. We recently made some changes and additions to the referential integrity rules.” Jeff added, “Also, you could structure this code a little bit differently, which would have given you more detail about the error.” Brian had the answers he needed, and thanked everyone for their help.

Next up was Pete, a front-end developer. “This is probably because I’m new to Bootstrap,” he said, “but I’m having a hard time getting the pending tasks page to display correctly on different size screens. The team looked over the code, then Howie spoke up. “I think that section right there is your problem. Those style classes you’ve added are fighting the framework instead of letting it do the work.” Howie showed some classes built into Bootstrap that Pete could be using instead. After some discussion and a few code changes, Pete had been put on the right path.

Lastly, Mahesh shared a problem he was having getting dates to display in the user’s local time zone in his JavaScript code. Jon excitedly put up his hand “I think I know what the problem is—I faced the same issue recently.” He proceeded to share what he had learned about quirks of the JavaScript date object and patterns he had found useful in converting between UTC time and local time. Mahesh soon had his issue resolved.

Jon felt terrific: knowledge he had learned from a past problem was now helping someone else. He reflected on how quickly these tough problems had been solved by a group effort. “We all know something about bugs and their causes,” he thought to himself, “but when we put our heads together our team is an amazing knowledge base.”

Epilogue

Evan was a young developer new to the project team. He had been told he needed to spend time with a lead developer who had an amazing reputation for quality. Evan couldn’t think of a subject more boring, and certainly didn’t feel he needed a mentor—but he was curious: if half the stories he heard were to be believed, this man could walk on water.

He entered the man’s office, a few minutes early, and sat down. There was a biography of Antonius Stradivarius on the desk, and a sign that said “The Bug Stops Here.” If the calendar on the wall was any indication, this man actually had free time for leisure activities. Despite himself, Evan started to become interested.

A minute later, a friendly man entered and shook Evan’s hand warmly. “Hello,” said the man. “My name is Jon.”

 

 

Advertisement