Custom Search

Friday, July 10, 2009

Priority and Severity of the Bug

Let’s peek in on a discussion in a bug triage meeting.

Tim, the marketing manager, is shaking his head. “That’s a high on the severity scale. It’s really bad, guys. You have to make it a high.”

Jordan, the development manager, is barely containing her frustration. Her eye is starting to twitch as she replies, “No, Tim. That’s not all that bad. It’s an inconvenience, I agree, but there’s an easy workaround.”

“Inconvenience?!?” Tim says a bit more loudly than he intended. “You call not being able to print an inconvenience?!? That’s a disaster!”
“Yes, I call not being able to print from one particular type of printer without installing an upgraded driver from the vendor’s website an inconvenience. The user just needs…”

“I know what the user needs,” Tim cut in. “The user needs to be able to print out of the box! You can fix this in our code, right?”

Jordan nods, “Yes, but we’d just be working around the vendor’s…”

“Then fix it.” Tim stood over Jordan, glaring.

“But it’s a medium at best!” Jordan objected. “The user isn’t losing any data, doesn’t have to reboot, isn’t crashing. They just have to update a driver.”

This argument could continue forever. I’ve seen many arguments like this go on and on. What’s really happening here? Why are Tim and Jordan about to be at each other’s throats?


Priority is Business; Severity is Technical
Tim is looking at business priority:

“How important is it to the business that we fix the bug?” Jordan is looking at technical severity: “How nasty is the bug from a technical perspective?” These two questions sometimes arrive at the same answer: a high severity bug is often also high priority, but not always. Allow me to suggest some definitions.
Severity is levels:
* Critical: the software

will not run
* High: unexpected fatal errors (includes crashes and data corruption)
* Medium: a feature is malfunctioning
* Low: a cosmetic issue

Now you see why Jordan was arguing that the Print bug was a medium: a feature was malfunctioning.

Priority levels:
* Now: drop everything and take care of it as soon as you see this (usually for blocking bugs)
* P1: fix before next build to test
* P2: fix before final release
* P3: we probably won’t get to these, but we want to track them anyway

And now you can see why Tim was so adamant that the issue was a high. From his perspective, it was a P1 matter.

They’re both right. It’s of medium severity, but P1 to fix.


Priority and Severity Don’t Mix
In response to Johanna’s column last week, some people suggested using both severity and priority to come up with a composite risk number.

While this intuitively sounds like a way to resolve the priority-severity divide, I suggest using such an approach with extreme caution. It’s multiplying apples by oranges in an attempt to quantify bananas. Risk is yet a third type of information.

The risk associated with any bug depends on the severity of the issue, certainly. But it also depends on the likelihood that the user will run into it as well as the possible losses that might occur. I don’t attempt to quantify all this when assessing the severity of an issue. In fact, I think that in most cases assessing the risk of a single issue takes more time than it’s worth. Only for potentially poisonous bugs involving dangerous fixes do I really want to weigh the risk of fixing it against the risk of not fixing it.

Establish Work Precedence
The best way to avoid confusion about what comes first is to ensure everyone in the organization takes their cues for work precedence from priority and nowhere else. Developers fix P1 defects first. Testers verify P1 fixes first. Technical writers document P1 issues first. Everyone works in priority order: the priority reflects importance to the business. Saying, “This bug is more severe than that one so I’ll work on it first” is as bad as saying, “I like this bug more, so I’ll work on it first.” The severity rating is technical information used by managers as a piece of the formula in determining the priority rating. The priority rating is the final word on the order in which the work is done by programmers, testers, and everyone else.

The ultimate lesson here, regardless of the terms or levels you use to categorize your bugs, is that any classification scheme will only be effective if everyone agrees on definitions. So perhaps that’s the very first question to ask when an argument is brewing about severity, priority, or risk: “Help me understand exactly what information you’re using from each defect record and how you’re using it?”

Thursday, July 9, 2009

Software Bugs And Spec Bugs

Let's assume that the error message wasn't displayed. In this case, we have a classic case of a software bug; i.e., a bug born because of a mismatch between the actual behavior of the software and the expected behavior (which is usually specified in the spec).

If you paid a little bit of attention when you read item 2.1., you surely noticed (joke) that it wasn't clear what kind of error message is expected to be displayed-i.e., the decision on the actual text of error message is up to the programmer, and he or she can write code that will produce:

- a NONinformative, irritating message: "Error", leaving the user to wonder what he or she has done wrong and cultivating a feeling of guilt in him or her, OR

- an easy-to-understand message: "Oops, error on page. ZIP code should have 5 digits".

Formally, the programmer will be right in both cases, because the specification doesn't elaborate about the text of the error message.


In early days of start-up when software changes very often, there is no need
to include precise text of error messages into the spec. The idea about error
message will do. For example: "Error message should state that ZIP code should
consist of 5 digits".

Here we have a situation where the spec itself is buggy, because

- we reasonably expect that spec to give us details about the error message, and

- the actual spec doesn't give us those details.

Let's call this situation "spec bug."

Brain positioning

Each found bug (both in software and in specs) must be filed into the bug tracking system.

There should be no exceptions. Bug reporting is one of the most important parts of our profession.

We'll have a separate lecture to talk about bug reporting and tracking.

Brain positioning

From the start, you must understand one very important thing: when

you file a bug against someone, it doesn't mean that you hate

or disrespect that person. It simply means: "I have found mismatch

between actual and expected. Please, fix the problem."

If somebody takes a bug personally, you must have the respect and

patience to explain to him or her that there is nothing personal here -

this is just a part of your job.


Three Conditions Of A Bug's Existence

A bug exists only if ALL of the following three conditions are met:

1. We know the expected result

2. We know the actual result

3. We know that the actual result deviates from the expected result


Every time you see a mismatch between the actual and expected, label that situation with the word "bug." As time goes on, you'll develop a habit, which will gradually grow into a reflex. Please note that for the sake of this training, it doesn't matter how futile, cheap, or temporary your expectations are - the main thing here is to gain the skill of automatic bug recognition.

Example

Here are some more bugs from real life:

1. A nice face backed by an ugly personality.

2. Parrots say the nastiest words at absolutely inappropriate times.

3. Hollywood story: reach fame and drink yourself to rehab.


Wednesday, July 8, 2009

The Big Bang Model


The Big Bang is built upon three main tenets:
  1. the universe used to be very hot
  2. the universe used to be very dense
  3. the universe is expanding (which is why it isn't so hot or dense anymore)

Note that the basic Big Bang Model does NOT say anything about the following questions:

  • will the universe collapse again, or expand forever?
  • is space curved or flat?
  • how old is the universe?
  • what is the matter density in the universe?
  • what about dark matter?
  • is there some mysterious "repulsive" force on large scales?
  • how did galaxies form?

Some of these questions all depend upon the values of certain parameters in the model, which we may derive from observations. Others have nothing to do with the Big Bang itself.

Our understanding of the laws of nature permit us to track the physical state of the universe back to a certain point, when the density and temperature were REALLY high. Beyond that point, we don't know exactly how matter and radiation behave. Let's call that moment the starting point. It doesn't mean that the universe "began" at that time, it just means that we don't know what happened before that point.

One of the primary successes of the Big Bang theory is its explanation for the chemical composition of the universe. Recall that the universe is mostly hydrogen and helium, with very small amounts of heavier elements. How does this relate to the Big Bang?

Well, a long time ago, the universe was hot and dense. When the temperature is high enough (a few thousand degrees), atoms lose all their electrons; we call this state of matter, a mix of nuclei and electrons, a fully-ionized plasma. If the temperature is even higher (millions of degrees), then the nuclei break up into fundamental particles, and one is left with a "soup" of fundamental particles:

  • protons
  • neutrons
  • electrons

Now, if the "soup" is very dense, then these particles will collide with each other frequently. Occasionally, groups of protons and neutrons will stick together to form nuclei of light elements ... but under extremely high pressure and temperature, the nuclei are broken up by subsequent collisions. The Big Bang theory postulates that the entire universe was so hot at one time that it was filled with this proton-neutron-electron "soup."

But the Big Bang theory then states that, as the universe expanded, both the density and temperature dropped. As the temperature and density fell, collisions between particles became less violent, and less frequent. There was a brief "window of opportunity" when protons and neutrons could collide hard enough to stick together and form light nuclei, yet not suffer so many subsequent collisions that the nuclei would be destroyed. This "window" appeared about three minutes after the starting point, and lasted for a bit less than a minute.

Which nuclei would form under these conditions? Experiments with particle colliders have shown us that most of the possible nuclei are unstable, meaning they break up all by themselves, or fragile, meaning they are easily broken by collisions





Helium (the ordinary sort, with 2 protons and 2 neutrons) is by far the most stable and robust compound nucleus. Deuterium (one proton and one neutron) is easily destroyed, and so is helium-3 (2 protons, one neutron).

So, it seems that this period of hot, dense plasma would create a lot of helium. Could it create other, heavier elements, too?






Detailed models of Big Bang nucleosynthesis predict that the brief "window of opportunity" lasted only a minute or two. After that, about three and a half minutes after the starting point, the temperature and density dropped so much that collisions between particles were rare, and of such low energy that the electric forces of repulsion between positively-charged nuclei prevented fusion. The result is

  • lots of hydrogen
  • some helium (ordinary helium-4)
  • tiny bits of deuterium
  • tiny bits of lithium
  • not much else


Friday, July 3, 2009

Testing Plan Template

Project Overview

(Optional) In this section, briefly discuss the project background and objectives. The appropriate information can be copied from the Project Definition document.

Testing Risks

Testing risks are circumstances and events that may occur before or during testing that would have an adverse impact on the testing process. For each high- and medium-level risk, determine what activities you will do to ensure that the risk does not occur. These risk plan activities should also be moved to the project workplan.

Testing Risk

Level (H/M/L)

Risk Plan

Example

The team is not familiar with the Web testing tools we will be using on this project.

H

· Try to find at least one person who has used the tools before.

· Send at least two team members to formal training.

· Set up follow-up training sessions to cross-train the rest of the developers.

Testing Schedule

This section contains a summary of the overall testing schedule. It is not at the activity level, but it should contain more detail than the major milestones included as a part of the Testing Strategy.

Testing Milestone/Events

Date Started

Date Completed

Example

Testing Strategy

05/15/YYYY

05/31/YYYY

Testing Plan

06/01/YYYY

06/30/YYYY

Unit Testing

08/15/YYYY

10/10/YYYY

—Test scripts turned in to project manager

10/10/XXXX

Integration Testing

—Integration coordinators assigned

09/15/YYYY

09/23/YYYY

—Integration testing

10/01/YYYY

11/01/YYYY

—Integration tests approved and scripts turned in

10/20/YYYY

11/01/YYYY


Unit Testing Design

Unit testing is the first and most basic level of testing. The testing focuses on individual components and is usually performed by the person who developed the component.

Person Responsible

Define the person responsible for the unit testing.

Person(s) Approving

Define the person(s) or organization responsible for approving individual unit tests and for validating that the component is ready to be moved to the next level of testing.

Test Environment

Describe any noteworthy features of the unit test environment, including location, equipment needed, software or tools used, etc.

Testing Process and Validation

Describe the testing logistics in some detail. This includes the types of tests to be performed, what you are trying to validate, how you will know if you are successful, how you will report errors, how the errors will be retested, etc. (A formal validation may not be needed for unit testing.)

Forms Used

Check the TechRepublic download library for unit test templates available in the future.


Integration Testing Design

Integration testing involves combining the individual components into the complete solution. The purpose is to test the interfaces between the various components and to ensure that the entire solution works as a whole according to the business requirements.

Person Responsible

Define the person responsible for the integration testing.

Person(s) Approving

Define the person(s) or organization responsible for approving the integration test and for validating that the group of components is ready to be moved to the next level of testing.

Test Environment

Describe any noteworthy features of the integration test environment, including location, equipment needed, software or tools used, etc.

Testing Process and Validation

Describe the testing logistics in some detail. This includes the types of tests to be performed, what you are trying to validate, how you will know if you are successful, how you will report errors, how the errors will be retested, etc.

Forms Used

Check the TechRepublic download library for integration test templates available in the future.


System Testing Design

System testing is the most important and complex of the testing stages and can consist of multiple subtests. System testing validates the overall requirements of the solution in terms of quality, performance, and functionality. The system test may include subtests such as the following:

· Requirements validation ensures that the solution satisfies the documented business requirements.

· Stress testing proves that the solution can handle heavy volumes of data, transactions, concurrent users, etc.

· Security testing ensures that the proper level of security and internal controls are in place.

· Disaster-recovery testing shows the solution’s ability to be recovered in case of disaster or massive failure.

· Usability testing determines how well the user will be able to use and understand the application. It identifies areas of poor human factors design that may make the system difficult to use.

· Procedures validation determines whether the user procedures and overall documentation is accurate and understandable.

Person Responsible

Define the person or group responsible for the system testing.

Person(s) Approving

Define the person(s) or organization responsible for approving the system testing and for validating that the solution is ready to be moved to the next level of testing.

Test Environment

Describe any noteworthy features of the system test environment, including location, equipment needed, software or tools used, etc. Many tools, processes, and locations may be needed for system testing that will not be used elsewhere on the project.

Testing Process and Validation

Describe the testing logistics in some detail. This includes the types of tests to be performed, what you are trying to validate, how you will know if you are successful, how you will report errors, how the errors will be retested, etc.

Forms Used

Check the TechRepublic download library for system test templates available in the future.


Acceptance Testing Design

Acceptance testing is designed for the customer to prove that the solution meets the original business requirement. In many cases, the customer is responsible for this testing since when the solution is completed successfully, it is expected that the customer will approve and accept it. Customers need to test the solution as they would expect to use it in production and then work with the project team to validate that the results come out as expected.

Person Responsible

Define the person or group responsible for the acceptance testing.

Person(s) Approving

Define the person(s) or organization responsible for approving the acceptance testing and for validating that the solution is ready to be moved into production. This level of approval should signify customer acceptance of the completed solution.

Test Environment

Describe any noteworthy features of the acceptance test environment, including location, equipment needed, software or tools used, etc.

Testing Process and Validation

Describe the testing logistics in some detail. This includes the types of tests to be performed, what you are trying to validate, how you will know if you are successful, how you will report errors, how the errors will be retested, etc.

Forms Used

Check the TechRepublic download library for acceptance test templates available in the future.

Optional Sections

You can add the following sections to the Testing Strategy if they provide more perspective and value to the reader or if the information is needed within your company.

Testing Methodology

Describe any specific methodology or standards that will help the reader understand the rationale behind each part of the Testing Plan.

Testing Assumptions

Document any assumptions surrounding the testing plan.

· Assumption #1

· Assumption #2

· Assumption #3

· Etc.

Testing Metrics

Describe any metrics you will capture as part of the testing process, such as total components tested, total defects per testing event, average time for defect correction, total number of hours spent on testing, total cost of testing, etc.