- June 11, 2020
Software Fault Injection
Executive Summary 2
Compile-time injections 4
Mutation testing 4
Code insertion 6
Practical examples 7
Thisis an expository paper on software fault injection. It elaborates thesignificance fault injection unlike conventional software testing.The main reason behind fault injection is to cut on the seriouslyhuge costs that are spent annually on debugging budget. This paperspecifically covers compile-time injection and the role it plays inthe software testing process. It further exposes mutation testing,code insertion and limitations of compile time injections. Practicalapplications of this technique are covered as well as how it relatesto company profitability.
Softwareerrors or bugs are so significant since they cost as estimate of$59.5 billion per year, which stands at about 0.6% of the overall GDPof the United States economy. National Institute of Standards andTechnology through the Department of Commerce have released thesedepressing statistics. Unfortunately, above half of the overall costsare met by the end users of the application and the remaining portionare met by either software vendors or developers (NationalInstitute of Standards and Technology, 2016).The study further revealed that, despite the fact that not all bugsmay be removed, but an estimate of about a third of the overall costcould be avoided through thorough software testing and debugging.Unfortunately, some bugs are not realized until too late in theapplication development process or even using end user experience(Voas, 2010). The eventual impact as a reflex of software bugs ishuge since virtually the business fraternity in the United Statesrelies totally on software for distribution, development, support ofafter-sales, and production of products. Failures of software areattracting a lot of attention currently due to their ubiquity andunpredictability. A perfect example is the recent launch failures ofthree Tritan rockets costing over $ 3B. The Air Force of the U.Sagreed that one such failure was because of an erratic decimal pointin the program (Voas, 2011).
Itwas established that software bugs cost the global economy a whopping$312 billion dollars per year out of the total $1.25 trillion dollarcost of software development. It also found that a programmer spendsabout half of his or her programming time rectifying such problems. Timely discovery of software bugs is crucial because the cost of abug can grow exponentially the further it persists down the softwaredevelopment life cycle (Britton,2013). The system of a large American trading firm called Knight CapitalGroup paid a huge price in 2012 when a code change was not deployedto just one of its many servers. This resulted in the erroneoussending of 4 million stock orders within an hour rather than theintended several days. This cost the company 4 times what it made inthe prior fiscal year at $440 million (Leon,2015).To resolve the complexity, new techniques need to be derived tocorrect errors in early stages of development process. The techniquediscussed in this paper is referred to as software fault injection.
Afault is a concept used to mean a physical imperfection or defectthat takes place within the software execution. In the event a faultyields an incorrect change during software execution, an errorappears. A fault may be localized in the given code but multipleruntime errors may originate from the fault source and saturate theentire application. The approach that we will use to establish theexistence of faults is software fault injection. The basic goal ofthis is to trip up application while running to determine itsreaction. Fault injection is rather empirical in nature. Softwarefault injection ideally should be carried out before release of theapplication and closer to the end of the development process. It isnot necessary to enforce fault injection when the application isshowing huge signs of defaults. It is rather advisable to be deployedonce the source code is stable and seemingly performing what it wasmeant for. A perfect example would be an addition operator in sourcecode (i.e. a=b+c) while leading to the user translating that as asubtraction operator (i.e. a=b-c) which in turn gives varying resultswhere, a= value. There are different scenarios of software injectionfaults, which are defined by the injection time such as runtime,compile-time, etc. In this paper, we will endeavor to exploresoftware fault injections carried out during compile-time.
Inorder for one to do injection of faults at this time, the source codemust be altered before the code is executed. This approach majors oninjecting bugs into the target application source code to simulatethe effect of the software faults. The altered source code changesthe target application instructions yielding injection. Injectiondevelops a tangible software image and thus when the targetapplication executes then it activates the anticipated fault.Essentially, injection in this manner develops an erroneous image ofthe software, which in turn activates the fault after execution ofthe faulty image (Hsueh et. al. 1997). This approach calls formodification of the source code, which in turn evaluates the accruingfault effect, and thus requires no extra software in the course ofruntime. Additionally, it does not cause perturbation to the systembeing targeted during the execution process. Because of effectscaused by the fault is essentially hard-coded, software developerscan easily make use of it to zero-in on permanent faults.Thisis a simple technique to deploy but does not accept fault injectionas a separate workload application runs (Hsueh et. al. 1997).
Themain advantage, which comes with this method, is that it can bedeployed in simulation of both software and hardware faults. Thetechnique has been proven in simulation of faults in an application,which are relatively closely related in their natural form close tothe ones generated through faults in programming. Since thensimulated faults are written into executable codes, it is thenpossible to segregate permanent from transient system faults. Again,this injection technique stands out as easy to implement code (Hsuehet. al., 1997). On the other side, the technique does not comewithout some unique limitations. The main setbacks of this techniqueinclude first, the method requires original source code modificationand thus the original code must always be available to the teamcarrying out the testing, which may not always be the scenario forCOTS applications for instance. Again, since the original source codeis undergoing modification, it is also most likely undesired codefaults may be freshly introduced. Lastly, as an adverse result of theoriginal source code undergoing some alterations, this method cannotbe deployed in place of certification processes because the systembeing tested will eventually be a very different system to theproduct. Asimple example, which can be given to demonstrate this technique, maybe in the event of changing: x= x+1 to x=x-1 (Looker et.al, 2010).
Inmutation, testing is described as a structural technique of softwaretesting, which makes use of the code structure to determine theprocess of testing. In an advanced stage, it is described as theunderlying process of regenerating the original code in smallversions with an objective of removing source code redundancies. Inthe event such program ambiguities are not fixed, this may resultinto software failures in the future and can easily bypass thedebugging process undetected (Briand, 2010). In mutation testing,the software developer takes the source code and performs thoroughtests on the suites arising from the application using othertechniques of testing. In this case, a number of related programs arecreated referred to as mutants, each having a small variance from thesource code that is each performing a fault processing. A goodexample is doing a replacement of a multiplication operator by anaddition operator. The initial test data sample is the executed onthe derived mutants. In the event such test cases end up detectingmutants which are different, then in the scenario the mutants arereferred to as killed, and the entire process of testing is assumedadequate (Briand, 2010).
Amutant is considered live in two distinct scenarios, which are in theevent where the mutant equates the source code, that is identicalfunctionally but different when viewed syntactically. The otherscenario is where the set of test codes are unable to kill theprevalent mutant. In the second scenario, there is need for the testdata set to be augmented through addition of one or several new testdata sets in order to kill the mutant still alive. In the event, themutants’ generation is automated, mutation operators are used,which means predefined source code modification rules (Anderson,2011). This is to mean defined rules, which correspond to the faultmodel. A keen note should be taken during mutation operators’specification. In ideal scenario, we would wish the operators ofmutation are representative of and generates all fault types, whichare realistic and could occur in an ordinary practice. Consequently,operators defined in mutation keeps on changing depending on theprogramming language used, specification paradigms, and design,though in many cases there are overlaps. Generally, there are manymutation operators since they are supposed to manipulate all possiblevariations syntactic in nature of a program. It is also possible tomake use of mutants in a random sampling nature. It is assumed thatmutants stand to be good effectiveness indicators (Briand, 2010).
Formutation coverage, in the event of complete coverage, this equalizesto killing all mutants, which are detected to be non-equivalent alsoknown as random sample. The amount of the total coverage is alsoreferred to as mutation score. Essentially, we can stand to view eachparticular mutant as a standalone test requirement. The eventualmutants’ number involved is depended on the definition of thesoftware structure as well as the mutation operators. In ordinarycases, the mutants’ number tends to be many, even in the eventwhere the application is small hence strengthening the essence ofusing random sampling (Briand, 2010). While holding such anunderstanding, one may pose a question of what happens in the eventthat errors are more complex involving a number of statements. Inorder to address this concern, there are two assumptions often madeas follows: the first one is where it is assumed that the programmersare competent. This is to mean that once they write a given code, itshould run with minimum errors and test as almost correct. The secondassumption is that of coupling effect. In this assumption, test caseswhich eventually differentiate all programs differing from the rathercorrect one by seemingly simple errors is very sensitive that inturn, implicitly distinguish seemingly more complicated errors(Anderson, 2011).
Thereare several fundamental points to take note of regarding mutationtesting. Mutation testing serves as a measurement of the quality ofindividual test cases. There is a tool’s slogan, Jester thatimplies the JUnit test tester. This gives the presiding person doingthe testing with a clearer target that is mutants available to kill.Mutation testing may also implicate that particular types of faultsare mostly unlikely, meaning, the ones specified by the precedingfault model, since it is established that the corresponding test casewill not fail. In this technique, it forces the software developer toinspect thoroughly the source code and imagine of possible test datathat is likely to expose certain types of code faults. Again, it isvery intensive in respect to the level of computation involved, sincethe number of mutants generated is large (Briand, 2010).
Despitethe fact that, mutation testing being very intensive in nature, thereseveral benefits which accrue upon adopting this testing technique.These include the following: to start with, it exposes the whole setof bugs to the software developer. Again, it stands out as the mostpowerful technique of exposing hidden bugs, which are ratherdifficult to recognize through conventional techniques of programtesting. Next, tools of testing like Insure ++ gives us theopportunity to recognize source code defects through thestate-of-the-art technique. Moreover, software products, which havebeen tested using the technique they are eventually accepted more andthe customer satisfaction score is rated high since they portray lesserrors upon development completion (Briand, 2010). Lastly, themaintenance and debugging the application is eventually easier ifthis technique is deployed before program completion. Suppose anoriginal source code bares the following statement in its code: a = a+ 1 this statement may be mutated to the following: a = a + a + 1holding the assumption that has a value other than zero (Voas, 2011).
Inthis approach, program instructions are loaded to the targetapplication code that in turn allows occurrence of fault injectionbefore a certain instruction executes which is closely related tocode modification. On the contrary, to code modification, thistechnique performs fault injection in the process of runtime andintroduces new program instructions other than altering the sourcecode. Interestingly, in this technique, the fault injector may runconcurrently with the target program code and execute at user mode inplace of system mode. This technique is different from compile-timeinjection because it does not participate in modification of existingbut instead adds code, which is fault generating. The method stillexhibits some static component of the compile-time injection method.This is because the executing source code must undergo somemodification and it is not corrupted but instead added to theoriginal code (Hsueh et. al., 1997).
Thistechnique prompts several advantages as follows: to begin with, interms of implementation procedure, it is relatively easy to do.Secondly,this method executes on computer processors, which do not necessarilypossess the necessary debugging hardware for the purpose and makesuse of alternative runtime injection methods. Inaddition, code insertion does not necessarily require any specialaccess to certain internal memory protected areas. However, thistechnique does not go down without some disadvantage of its own. Themain evident demerit is that, it demands the application source codeto be altered and thus making it relatively difficult to deploy thismethod during system certification (Looker et.al. 2010).
Softwarefault injection does not come without its challenges and limitations.The following is a discussion of such limitations: To start with,there is limited number of the injection, which can be executed. Thisis because it is only carried out at the assembly instruction point.Secondly, this approach cannot perform software faults injection intolocations that are unreachable by the application. Again, it alsorequires an active modification of the original code instruction inorder to support the overall fault injection. This implies that, theprogram instructions running during the fault injection experimentsare not the particular ones, which executes upon release of thesoftware (Ziade,et. al., 2004).Next, this approach also poses limitation in controllability andobservability. At the best-case scenario, one is exposed tocorruption of the underlying registers of the internal processors andthe locations under the internal memory map. This is the only extendthe programmer is able to see which was traditionally known assoftware developer’s processor model. This implies that, faultinjection cannot be done in the processor scheduling or softwareinstruction pipeline (Ziade,et. al., 2004).
Moreover,on software fault injection, it is literally very difficult and hardto model faults, which are permanent. This stands out as a seriouslimitation since in such a scenario the software cannot be correctedin this approach. Lastly, while running the fault injection, theprogram instruction could have a ripple effect on the processorscheduling of all other tasks in the machine. The cause effect ofsuch a limitation is where system real- time deadlines being missed(Ziade,et. al., 2004).
A good practical example is the Ariane 5 rocket disaster. A welltested source code subjected through all fault injection proceduresworked normally and as expected in Ariane 4. This mistake done was toassume that since the source code worked in Ariane 4, and theneventually it would work in Ariane 5, which turned out a disastrouscrash (Voaset. al., 2010).Another good example of software fault injection is the developmentof software implementation of incumbent disk fault injection majorlydrifting its target to x86/Linux fraternity. Software fault injectionhas been deployed successfully in several other fields such asdefinition of processors, networking, and even kernels (Adriaens& Gibson, 2005).
Inconclusion, software faults result into heavy recovery costs if theyare not detected before the program is shipped. The heavy costsassociated with debugging a program while in use by the end usersmake it necessary to deploy software fault injection techniques toaddress the problem. This term paper has explored compile timeinjection as well as detailed mutation testing, code insertion aswell as limitations from software fault injection and the respectivetechniques. The paper also has tried to express several practicalexamples of real life scenarios where fault injection has beenaccomplished with great success. In order to cut on associated costsof debugging process, software developers must invest on faultinjection to correct possible errors before releasing an applicationto the end users. With the exploration discussed through the paper,the world can realize great savings of the GDP if programmers becomeresponsible in this world of software dependency.
Adriaens,J., & Gibson, D (2005). A Software Layer for IDE Disk FaultInjection: To appear in Systems Lacking Originality Workshop 2005.Universityof Wisconsin-Madison Department of Electrical and ComputerEngineering.
Anderson,S. (2011). Mutation Testing. Schoolof Informatics.Retrieved fromhttp://www.inf.ed.ac.uk/teaching/courses/st/2011-12/Resource-folder/09_mutation.pdf
Briand,L. (2010). Software Verification and Validation. ResearchLaboratory.Retrieved fromhttp://www.inf.ed.ac.uk/teaching/courses/st/2015-16/Mutest.pdf
Britton,T., Jeng, L., Carver, G., Cheak, P. & Katzenellenbogen, T."Reversible debugging software." TechnicalReport (2013): 1-18. CiteSeerX. Cambridge University, 2013.Web. 23 Nov. 2016.
Hsueh,M., Tsai, T. & Iyer, R. (1997). Fault Injection Techniques andTools. Universityof Illinois at Urbana-Champaign,75- 82. Retrieved fromhttps://www.ece.cmu.edu/~ece749/docs/faultInjectionSurvey.pdf
Leon,J. (2015, February 28). The true cost of a software bug: part one.Retrieved November 23, 2016, fromhttp://blog.celerity.com/the-true-cost-of-a-software-bug
Looker,N., Munro, M., & Xu, J. (2010). AComparison of Network Level Fault Injection with Code Insertion.Retrieved from:https://pdfs.semanticscholar.org/ec18/00dd1278e87366c896de032994b57f95bfb5.pdf
NationalInstitute of Standards and Technology. (2016). Software Errors CostU.S. Economy $59.5 Billion Annually. NationalInstitute of Standards and Technology.Retrieved fromhttp://www.ashireporter.org/HomeInspection/Articles/Software-Errors-Cost-U-S-Economy-59-5-Billion-Annually/740
Voas,J. (2010). Soft are Hazard Mining. ReliableSoftware Technologies.Retrieved from https://www.cigital.com/papers/download/asset99.pdf
Voas,J. (2011). Tutorialon . ReliableSoftware Technologies.Retrieved fromhttps://www.cigital.com/papers/download/spectrum2000.pdf
Voas,J., McGraw, G., Kassab, L., & Voas, L. (2010). A Crystal Ball forSoftware Liability. Retrieved fromhttps://www.cigital.com/papers/download/crystal.pdf
Ziade,H., Ayoubi, R. & Velazco, R. (2004). A Survey on Fault InjectionTechniques. TheInternational Arab Journal of Information Technology,1(2), 171-187.