Unit V: Software Reliability - Software Engineering

Introduction to Software Reliability:

Software reliability is one of the most important elements of the overall quality of any software even after the accomplishment of software work. If it fails to meet its actual performance after its deployment, then the software is considered as unreliable software. We cannot expect better performance from such software.

Software reliability is defined in statistical terms as the probability of failure-free operation of a computer program in a specific environment for a specific time for a software to be reliable, it must perform operation based upon its analysis and design, available resources, reusability (if reusable components are involved) and so on.

If the design standards are not maintained, efficient algorithms are not implemented, necessary resources are not available or the supply is not at proper times, then there is a high probability of software failure or there is degradation in the quality of performance.

Therefore, in order to make software reliable, we should take some measures or earlier stages as follows:

1. Designing a software project based on available resources.

2. Develop software that best fits the current environmental conditions.

3. Estimating costs that might be required even after its implementation.

4. Estimating the actual performance upon using reusable resources.

Measures of Reliability and Availability:

People often confuse reliability and availability. Simply put availability is a measure of the % of time the equipment is in an operable state while reliability is a measure of how long the item performs its intended function. We can refine these definitions by considering the desired performance standards.

Reliability is a measure of the probability that an item will perform its intended function for a specified interval under stated conditions. 

There are two commonly used measures of reliability:

1. Mean Time Between Failure (MTBF), which is defined as, total time in service / number of failures

2. Failure Rate (λ), which is defined as, number of failures / total time in service.

A piece of equipment can be available but not reliable. For example, the machine is down 6 minutes every hour. This translates into availability of 90% but reliability of less than 1 hour. That may be okay in some circumstances but what if this is a paper machine? It will take at least 30 minutes of run time to get to the point that we are producing good paper.

Availability is an Operations parameter as, presumably, if the equipment is available 85% of the time, we are producing at 85% of the equipment's technical limit. This usually equates to the financial performance of the asset. Of course, quality and machine speed need to be considered in order to have a proper representation of how close we are to this technical limit. This is called OEE (Overall Equipment Effectiveness).

Availability can be measured as Uptime / Total time (Uptime + Downtime).

Generally speaking, a reliable machine has high availability but an available machine may or may not be very reliable.

Software Reliability Measurement Techniques:

Reliability metrics are used to quantitatively express the reliability of the software product. The option of which parameter is to be used depends upon the type of system to which it applies & the requirements of the application domain.

Measuring software reliability is a severe problem because we don't have a good understanding of the nature of software. It is difficult to find a suitable method to measure software reliability and most of the aspects connected to software reliability. Even the software estimates have no uniform definition. If we cannot measure the reliability directly, something can be measured that reflects the features related to reliability.

The Current Methods Of Software Reliability Measurement Can Be Divided Into Four Categories:

1. Product Metrics:

Product metrics are those which are used to build the artefacts, i.e., requirement specification documents, system design documents, etc. These metrics help in the assessment if the product is right sufficient through records on attributes like usability, reliability, maintainability & portability. In these measurements are taken from the actual body of the source code.

a. Software size is thought to be reflective of complexity, development effort, and reliability. Lines of Code (LOC), or LOC in thousands (KLOC), is an initial intuitive approach to measuring software size. The basis of LOC is that program length can be used as a predictor of program characteristics such as effort &ease of maintenance. It is a measure of the functional complexity of the program and is independent of the programming language.

b. Function point metric is a technique to measure the functionality of proposed software development based on the count of inputs, outputs, master files, inquires, and interfaces.

c. Test coverage metric size fault and reliability by performing tests on software products, assuming that software reliability is a function of the portion of software that is successfully verified or tested.

d. Complexity is directly linked to software reliability, so representing complexity is essential. Complexity-oriented metrics is a way of determining the complexity of a program's control structure by simplifying the code into a graphical representation. The representative metric is McCabe's Complexity Metric.

e. Quality metrics measure the quality at various steps of software product development. A vital quality metric is Defect Removal Efficiency (DRE). DRE provides a measure of quality because of different quality assurance and control activities applied throughout the development process.

2. Project Management Metrics:

Project metrics define project characteristics and execution. If there is proper management of the project by the programmer, then this helps us to achieve better products. A relationship exists between the development process and the ability to complete projects on time and within the desired quality objectives. Cost increase when developers use inadequate methods. Higher reliability can be achieved by using a better development process, risk management process, configuration management process.

These Metrics Are:

a. Number of software developers.

b. Staffing pattern over the life-cycle of the software.

c. Cost and schedule.

d. Productivity

3. Process Metrics:

Process metrics quantify useful attributes of the software development process & its environment. They tell if the process is functioning optimally as they report on characteristics like cycle time & rework time. The goal of process metric is to do the right job on the first time through the process. The quality of the product is a direct function of the process. So process metrics can be used to estimate, monitor, and improve the reliability and quality of software. Process metrics describe the effectiveness and quality of the processes that produce the software product.

Examples:

a. The effort required in the process

b. Time to produce the product

c. Effectiveness of defect removal during development

d. Number of defects found during testing

e. Maturity of the process

4. Fault and Failure Metrics:

A fault is a defect in a program which appears when the programmer makes an error and causes failure when executed under particular conditions. These metrics are used to determine the failure-free execution software.

To achieve this objective, a number of faults found during testing and the failures or other problems which are reported by the user after delivery are collected, summarized, and analyzed. Failure metrics are based upon customer information regarding faults found after release of the software. The failure data collected is therefore used to calculate failure density, Mean Time between Failures (MTBF), or other parameters to measure or predict software reliability.

Reliability Metrics:

Reliability metrics are used to quantitatively express the reliability of the software product. The option of which metric is to be used depends upon the type of system to which it applies & the requirements of the application domain.

Some reliability metrics which can be used to quantify the reliability of the software product are as follows:

1. Mean Time to Failure (MTTF):

MTTF is described as the time interval between the two successive failures. An MTTF of 200 means that one failure can be expected each 200-time units. The time units are entirely dependent on the system & it can even be stated in the number of transactions. MTTF is consistent for systems with large transactions.

For example, It is suitable for computer-aided design systems where a designer will work on a design for several hours as well as for Word-processor systems.

To measure MTTF, we can evidence the failure data for n failures. Let the failures appear at the time instants t₁,t₂.....t_n.

MTTF Can Be Calculated As:

2. Mean Time to Repair (MTTR):

Once failure occurs, some-time is required to fix the error. MTTR measures the average time it takes to track the errors causing the failure and to fix them.

3. Mean Time Between Failure (MTBR):

We can merge MTTF & MTTR metrics to get the MTBF metric.

MTBF = MTTF + MTTR

Thus, an MTBF of 300 denoted that once the failure appears, the next failure is expected to appear only after 300 hours. In this method, the time measurements are real-time & not the execution time as in MTTF.

4. Rate of Occurrence of Failure (ROCOF):

It is the number of failures appearing in a unit time interval. The number of unexpected events over a specific time of operation. ROCOF is the frequency of occurrence with which unexpected role is likely to appear. A ROCOF of 0.02 means that two failures are likely to occur in each 100 operational time unit steps. It is also called the failure intensity metric.

5. Probability of Failure on Demand (POFOD):

POFOD is described as the probability that the system will fail when a service is requested. It is the number of system deficiency given several systems inputs. POFOD is the possibility that the system will fail when a service request is made.

A POFOD of 0.1 means that one out of ten service requests may fail. POFOD is an essential measure for safety-critical systems. POFOD is relevant for protection systems where services are demanded occasionally.

6. Availability (AVAIL):

Availability is the probability that the system is applicable for use at a given time. It takes into account the repair time & the restart time for the system. An availability of 0.995 means that in every 1000 time units, the system is feasible to be available for 995 of these. The percentage of time that a system is applicable for use, taking into account planned and unplanned downtime. If a system is down an average of four hours out of 100 hours of operation, its AVAIL is 96%.

Software Reliability Models:

The basic goal of software engineering is to produce high-quality software at a low cost. With the growth in size and complexity of software, management issues began dominating. Reliability is one of the representative qualities of the software development process. Reliability of software is basically defined as the probability of expected operation over a specified time interval.

Software Reliability is an important attribute of software quality, together with functionality, usability, performance, serviceability, capability, installability, maintainability, and documentation. Reliability is the property of referring ‘how well the software meets its requirements’ & also ‘the probability of failure-free operation for the specified period of time in a specified environment. Software reliability defines as the failure-free operation of a computer program in a specified environment for a specified time. Software Reliability is hard to achieve because of the complexity of software tends to be high.

A proliferation of software reliability models have emerged as people try to understand the characteristics of how and why software fails, and try to quantify software reliability. Software Reliability Modelling techniques can be divided into two subcategories: Prediction modelling and Estimation modelling. Both kinds of modelling techniques are based on observing and accumulating failure data and analyzing with statistical inference.

1. Prediction Models:

This model uses historical data. They analyze previous data and some observations. They are usually made prior to development and regular test phases. The model follows the concept phase and the prediction from the future time.

a. Musa Model:

This prediction technique is used to predict, prior to system testing, what the failure rate will be at the start of system testing. This prediction can then later be used in the reliability growth modelling. For this prediction method, it is assumed that the only thing known about the hypothetical program is a prediction of its size and the processor speed.

This model assumes that the failure rate of the software is a function of the number of faults it contains and the operational profile. The number of faults is determined by multiplying the number of developed executable source instructions by the fault density. Developed excludes re-used code that is already debugged. Executable excludes data declarations and compiler directives. For fault density at the start of system test, a value for faults per KSLOC needs to be determined. For most projects, the value ranges between 1 and 10 faults per KSLOC. Some developments which use rigorous methods and highly advanced software development processes may be able to reduce this value to 0.1 fault/KSLOC.

b. Putnam Model:

Created by Lawrence Putnam, Sr. the Putnam model describes the time and effort required to finish a software project of specified size. SLIM (Software Lifecycle Management) is the name given by Putnam to the proprietary suite of tools his company QSM, Inc. has developed based on his model. It is one of the earliest of these types of models developed and is among the most widely used. Closely related software parametric models are Constructive Cost Model (COCOMO), Parametric Review of Information for Costing and Evaluation – Software (PRICE-S), and Software Evaluation and Estimation of Resources – Software Estimating Model (SEER-SEM).

Putnam’s observations about productivity levels to derive the software equation: 

Where:

1. Size is the product size (whatever size estimate is used by your organization is appropriate). Putnam uses ESLOC (Effective Source Lines of Code) throughout his books.

2. B is a scaling factor and is a function of the project size Productivity is the Process Productivity, the ability of a particular software organization to produce software of a given size at a particular defect rate.

3. Effort is the total effort applied to the project in person-years.

4. Time is the total schedule of the project in years.

In practical use, when making an estimate for a software task the software equation is solved for effort: 

An estimated software size at project completion and organizational process productivity is used. Plotting effort as a function of time yields the Time-Effort Curve. The points along the curve represent the estimated total effort to complete the project at some time. One of the distinguishing features of the Putnam model is that total effort decreases as the time to complete the project is extended. This is normally represented in other parametric models with a schedule relaxation parameter.

Thus estimating method is fairly sensitive to uncertainty in both size and process productivity. Putnam advocates obtaining process productivity by calibration.

c. Rome Lab TR-92-52 Model:

Software Reliability Measurement and Test Integration Techniques Method RL-TR-92-52 contain empirical data that was collected from a variety of sources, including the Software Engineering Laboratory. The model consists of 9 factors that are used to predict the fault density of the software application. The nine factors are:

There are certain parameters in this prediction model that have tradeoff capability. This means that there is a large difference between the maximum and minimum predicted values for that particular factor. Performing a tradeoff means that the analyst determines where some changes can be made in the software engineering process or product to experience an improved fault density prediction. A tradeoff is valuable only if the analyst has the capability to impact the software development process. The output of this model is a fault density in terms of faults per KSLOC. This can be used to compute the total estimated number of inherent defects by simply multiplying by the total predicted number of KSLOC.

d. Rayleigh Model:

This model predicts fault detection over the life of the software development effort and can be used in conjunction with other prediction techniques. Software management may use this profile to gauge the defect status. This model assumes that over the life of the project that the faults detected per month will resemble a Raleigh curve.

The Rayleigh model is the best suited model for estimating the number of defects logged throughout the testing process, depending on the stage when it was found (unit testing, integration testing, system testing, functional testing etc.). Defect prediction function is the following: 

F(t)ʄ(K, t, t)m = (1)

Parameter K is the cumulated defect density, t is the actual time unit and m is the time at the peak of the curve.

For example, the estimated number of defects impacts the height of the curve while the schedule impacts the length of the curve. If the actual defect curve is significantly different from the predicted curve then one or both of these parameters may have been estimated incorrectly and should be brought to the attention of management.

2. Estimation Model:

This model uses the current data from the current software development effort and doesn't use the conceptual development phases and can estimate at any time.

a. Weibull Model (WM):

This model is used for software/hardware reliability. The model incorporates both increasing/decreasing and failure rate due to high flexibility. This model is a finite failure model. 

MTTF = ∫(1 – F(t)) = ∫exp (-þtα)

Where, MTTF = Mean Time to Failure

α, þ = Weibull distribution parameters.

t = Time of failure.

A Weibull-type testing-effort function with multiple change-points can be estimated by using the methods of least squares estimation (LSE) and maximum likelihood estimation (MLE) The LSE minimizes the sum of squares of the derivations between actual data patterns and predicted curve, while the MLE estimates parameters by solving a set of simultaneous equations.

b. Bayesian Models (BM):

The models are used by several organizations like Motorola, Siemens & Philips for predicting reliability of software. BM incorporates past and current data. Prediction is done on the bases of a number of faults that have been found & the amount of failure-free operations. The Bayesian SRGM considers reliability growth in the context of both the number of faults that have been detected and the failure-free operation. Further, in the absence of failure data, Bayesian models consider that the model parameters have a prior distribution, which reflects a judgment on the unknown data based on history e.g. a prior version and perhaps expert opinion about the software.

c. J -M Model (JMM):

This model assumes that failure time is proportional to the remaining faults and taken as an exponential distribution. During the testing phase the number of failures at first is finite. Concurrent mitigation of errors is the main strength of the model and error does not affect the remaining errors. Error removal is all human behaviour which is irregular so it cannot be avoided by introducing new errors during the process of error removal.

(MTBF)t = 1 / (N – (i - 1))

Where, N = Total Number of Faults.

i = Number of fault occurrences.

MTBF = Mean Time Between Failure

t = Time Between The Occurrence of the (i – 1)^st and i^th fault occurrences.

This model assumes that N initial faults in the code prior to testing are a fixed but known value. Failures are not correlated and the times between failures are independent and exponentially distributed random variables. Fault removal on failure occurrences is instantaneous and does not introduce any new faults into the software under test. The hazard rate z(t) of each fault is time-invariant and a constant. Moreover, each fault is equally likely to cause failure.

d. Goel-Okumoto Model (GOM):

G-O model takes the number of faults per unit time as independent random variables. In this model, the number of faults occurred within the time and model estimates the failure time. Delivery of software within cost estimates is also decided by this model.

The Goel-Okumoto (G-O) non-homogeneous Poisson process (NHPP) model has slightly different assumptions from the J-M model. The significant difference between the two is the assumption that the expected number of failures observed by time t follows a Poisson distribution with abounded and non-decreasing mean value function (t). The expected value of the total number of failures observed in infinite time is a finite value N.

e. Thompson and Chelson’s Model:

In this model the number of failures detected in each interval (fi) and Length of testing time for each interval i(Ti). 

(fi + f0 + 1) / (Ti + T0)

Software is corrected at the end of the testing interval. Software is operational. Software is relatively faulted free. The Thompson Chelson model can be used, If the failure intensity is decreasing, has the software been tested or used in an operational environment representative of its end usage, with no failures for a significant period of time.

Software Safety:

Software safety is an SQA activity that focuses on the identification and assessment of potential hazards that may affect software negatively and cause an entire system to fail. If hazards can be identified early in the software process, software design features can be specified that will either eliminate or control potential hazards.

A modelling and analysis process is conducted as part of software safety. Initially, hazards are identified and categorized by critically and risk. For example, some of the hazards associated with a computer base cruise control for an automobile might be:

1. Causes uncontrolled acceleration that cannot be stopped.

2. Does not respond to depression of brake pedal (by turning off)

3. Does not engage when the switch is activated.

4. Slowly loses or gain speed.

Once such hazards are identified analysis techniques are used to assign severity and probability of occurrence.

Software reliability uses statistical analysis to determine the likelihood that a software failure will cause. However, the occurrence of failure does not necessarily result in a mishap (accident). Software safety examines the ways in which failure result in conditions that can lead to a mishap. That is, failures are not considered in a vacuum but are evaluated in the context of an entire computer-based system and its environment.