Web Testing |
|
| Web Testing |
|
|
|
| Written by Administrator | |
| Saturday, 06 May 2006 | |
|
WEB TESTING Why we do web testing? Most web sites and web applications run smoothly and correctly as long as only one user (e.g. the developer) is using it. But what happens when thousands of users access the website simultaneously?! The instant worldwide audience of any Web Browser Enabled Application -- a Web site -- makes its quality and reliability crucial factors in its success. Web Sites impose some entirely new challenges in the world of software quality! Within minutes of going live, a Web application can have many thousands more users than a conventional, non-Web application. The immediacy of the Web creates immediate expectations of quality and rapid application delivery, but the technical complexities of a Web Site and variances in the browser make testing and quality control that much more difficult, and in some ways, more subtle, than "conventional" client/server or application testing. Automated testing of Web Sites is an opportunity and a challenge. The Business View of web testing Today many websites have a serious business background and are mainly run to sell a service or a product to make money. Everybody running a business on the web has seen statistics how fast a customer clicks away to your competitor if your server reacts too slow or not at all. Aren't you cancelling the checkout of a shopping cart if the webserver does not react within 5-20 seconds?! If the visitors to your web site know about your slow website before you do, it may cost you a lot of money! For the owner of a website this means: Test and monitor your website! Not only functionality testing (does the website do what it is meant to do) and usability testing (does the user understand what is going on in his browser window) but also performance testing (does the user get results in an acceptable time) is important. Your goal must be to ensure your customer always gets the right answer to his mouseclick within a few seconds at anytime! Ensure that 95% of requests are processed in less than 10 seconds. The Technical View of web testing Moving the focus of our thoughts from the business to the technical viewpoint we discover the following questions that need to be answered to ensure that the goals from the business viewpoint are met: Is your webserver prepared for as many users as you expect?! Is your webserver prepared for increasing load over the month/years? Does your webserver survive a massive peak of users (e.g. if your website is mentioned on national TV) How many users can your server handle before users will get error messages or timeouts? How long does it take for a visitor to receive a page under no load, little load and heavy load?! Does your application or shopping cart support simultaneous users at all?! Are the CGIs, databases or scripts programmed as fast as possible and do they interact with each other correctly under heavy load?! Is the hosting service doing a good job?! Is the internet connection bandwidth sufficient?! Is that expensive hardware too small or too powerful?! You need to find out all of these answers! Tests in Web Testing Performance-, Load- or Stress-Testing These three words are often used synonimously, but there are differences: Performance-Tests: Are used to test each part of the webserver or the web application to find out what parts of the website are slow and how you can make them faster. Think of it as testing of single webpages/scripts. Webserver Stress Tool supports this type of test by the ability to running some 20-40 simultaneous requests on one url and recording the average time until the requests are answered. By changing the programming code you will be able to find the points you have to take care of to gain more website performance. Usually this type of test is run without requesting all the images etc. in a webpage to get more exact results about the script processing Load-Tests : Are done by testing the website using the load that you expect to have on your site. This is something like a real world test of the website. First you have to define the maximum request times you want the customers to experience, this is done from the business and usability point of view, not from a technical point of view. At this point you need to calculate the impact of a slow website on your company sales and support costs. Then you have to calculate the anticipated load and load pattern for your website (see next section for details on load calculation!) which you then simulate using Webserver Stress Tool. At the end you compare the test results with the requests times you wanted to achieve. You know you have some work to do when requests take longer then time out or even generate error messages! Stress-Tests: Are simulated brute force attacks with excessive load on your webserver. In the real world situations like this can be created by a massive spike of users - far above the normale usage - e.g. caused by a large referrer (imagine your website being mentioned on national TV). The goals of stress tests are to learn under what load your server generates errors, whether it will come back online after such a massive spike at all or crash and when it will come back online. Calculation of load and load pattern This is probably the trickiest question in conducting performance tests on websites. First, remember that there is a difference between users, transactions, pageviews and hits: One user creates several (trans-)actions (e.g. visit Homepage, search product, view product's details, buy a product etc.) One (trans-)action creates several pageviews (e.g. add products to shopping cart, go to checkout, enter creditcard etc.) One pageview creates several hits (e.g. framesets, images, applets etc. of one webpage) If you already have your website online it is a good start to use a good logfileanalyzer on your logfiles. You can find out how many people access the site per day and per hour, what pages/scripts are used how often etc. If you are working on a new website you have to find the numbers and pattern yourself. One way to define the load pattern could be: Step 1: From a business point of view come up with the target number of users. Step 2: Define a couple of different "model users" (e.g. teen kid, mature business customer, senior citizen etc.) and surf from their point of view through the website. Track the webpages they access and gather these stats. At the end you should have a list of URLs and their frequency of use. There is a difference between the number of users using your website at one time and users sending simultaneous requests to your website! If you have 200 simultaneous users clicking a link every 20 seconds you have 600 clicks per minute (3 per user per minute) or 10 simultaneous users per second. This number would be the number you simulate with Webserver Stress Tool. Remember to take into account that additionally to the average load of your website there could be spikes in usage generated by marketing activities (e.g. newsletters, banners, TV commercials etc.) or simply by seasonal situations (e.g. Valentines Day for a flower shop website). When does testing start? The answer is simple: You cannot start performance testing early enough when building web applications! It is even a good idea to start performance testing before a line of code is written at all! Early testing the base technology (network, load balancer, application-, database- and web-servers) for the load levels you plan for can save a lot of money when you can already discover at this moment that your hardware is to slow. Also the first stress tests can be a good idea at this point. The costs for correcting a performance problem rise steeply from the start of development until the website goes productive and can be unbelievable high for a website already online. During software development all programmers should have access to performance test tools from an early stage on to test their own code for performance and for parallel execution problems (e.g. caused by database locks or other mutexes). Each developer should be responsible not only for the functionality of his code but also for good performance of his pages! As soon as several webpages are working the first load tests should be conducted and from there on should be part of the regular testing routine each day or week or for each build of the software. Defining Website Quality & Reliability Like any complex piece of software there is no single, all-inclusive quality measure that fully characterizes a Web Site (by which we mean any web browser enabled application). Dimensions of Quality; There are many dimensions of quality; each measure will pertain to a particular Web Site in varying degrees. Here are some common measures: There are many dimensions of quality; each measure will pertain to a particular Web Site in varying degrees. Here are some common measures:Timeliness: Web Sites change often and rapidly. How much has a Web Site changed since the last upgrade? How do you highlight the parts that have changed? Web Sites change often and rapidly. How much has a Web Site changed since the last upgrade? How do you highlight the parts that have changed?Structural Quality: How well do all of the parts of the Web Site hold together? Are all links inside and outside the Web Site working? Do all of the images work? Are their parts of the Web Site that are not connected? How well do all of the parts of the Web Site hold together? Are all links inside and outside the Web Site working? Do all of the images work? Are their parts of the Web Site that are not connected?Content: Does the content of critical pages match what is supposed to be there? Do key phrases exist continually in highly changeable pages? Do critical pages maintain quality content from version to version? What about dynamically generated HTML (DHTML) pages? Does the content of critical pages match what is supposed to be there? Do key phrases exist continually in highly changeable pages? Do critical pages maintain quality content from version to version? What about dynamically generated HTML (DHTML) pages?Accuracy and Consistency: Are today's copies of the pages downloaded the same as yesterday's? Close enough? Is the data presented to the user accurate enough? How do you know? Are today's copies of the pages downloaded the same as yesterday's? Close enough? Is the data presented to the user accurate enough? How do you know?Response Time and Latency: Does the Web Site server respond to a browser request within certain performance parameters? In an e-commerce context, how is the end-to-end response time after a SUBMIT? Are their parts of a site that are so slow the user discontinue working? Does the Web Site server respond to a browser request within certain performance parameters? In an e-commerce context, how is the end-to-end response time after a SUBMIT? Are their parts of a site that are so slow the user discontinue working?Performance: Is the Browser --> Web -->Web Site --> Web --> Browser connection quick enough? How does the performance vary by time of day, by load and usage? Is performance adequate for e-commerce applications? Taking 10 minutes -- or maybe even only 1 minute -- to respond to an e-commerce purchase may be unacceptable! Is the Browser --> Web -->Web Site --> Web --> Browser connection quick enough? How does the performance vary by time of day, by load and usage? Is performance adequate for e-commerce applications? Taking 10 minutes -- or maybe even only 1 minute -- to respond to an e-commerce purchase may be unacceptable!Impact of Quality, Quality remains is in the mind of the Web Site user. A poor quality Web Site, one with many broken pages and faulty images, with Cgi-Bin error messages, etc., may cost a lot in poor customer relations, lost corporate image, and even in lost sales revenue. Very complex, disorganized Web Sites can sometimes overload the user. Quality remains is in the mind of the Web Site user. A poor quality Web Site, one with many broken pages and faulty images, with Cgi-Bin error messages, etc., may cost a lot in poor customer relations, lost corporate image, and even in lost sales revenue. Very complex, disorganized Web Sites can sometimes overload the user.The combination of Web Site complexity and low quality is potentially lethal to Company goals. Unhappy users will quickly depart for a different site; and, they probably won't leave with a good impression. Website Architectural Factors A Web Site can be quite complex, and that complexity -- which is what provides the power, of course -- can be a real impediment in assuring Web Site Quality. Add in the possibilities of multiple Web Site page authors, very-rapid updates and changes, and the problem compounds. Here are the major pieces of Web Sites as seen from a Quality perspective. Browser. The browser is the viewer of a Web Site and there are so many different browsers and browser options that a well-done Web Site is probably designed to look good on as many browsers as possible. This imposes a kind of de facto standard: the Web Site must use only those constructs that work with the majority of browsers. But this still leaves room for a lot of creativity, and a range of technical difficulties. And, multiple browsers' renderings and responses to a Web Site have to be checked. The browser is the viewer of a Web Site and there are so many different browsers and browser options that a well-done Web Site is probably designed to look good on as many browsers as possible. This imposes a kind of de facto standard: the Web Site must use only those constructs that work with the majority of browsers. But this still leaves room for a lot of creativity, and a range of technical difficulties. And, multiple browsers' renderings and responses to a Web Site have to be checked.Display Technologies. What you see in your browser is actually composed from many sources: What you see in your browser is actually composed from many sources:HTML. There are various versions of HTML supported, and the Web Site ought to be built in a version of HTML that is compatible. This should be checkable. There are various versions of HTML supported, and the Web Site ought to be built in a version of HTML that is compatible. This should be checkable.Java, JavaScript, ActiveX. Obviously JavaScript and Java applets will be part of any serious Web Site, so the quality process must be able to support these. On the Windows side, ActiveX controls have to be handled well. Obviously JavaScript and Java applets will be part of any serious Web Site, so the quality process must be able to support these. On the Windows side, ActiveX controls have to be handled well.Cgi-Bin Scripts. This is link from a user action of some kind (typically, from a FORM passage or otherwise directly from the HTML, and possibly also from within a Java applet). All of the different types of Cgi-Bin Scripts (perl, awk, shell-scripts, etc.) need to be handled, and tests need to check "end to end" operation. This kind of a "loop" check is crucial for e-commerce situations. This is link from a user action of some kind (typically, from a FORM passage or otherwise directly from the HTML, and possibly also from within a Java applet). All of the different types of Cgi-Bin Scripts (perl, awk, shell-scripts, etc.) need to be handled, and tests need to check "end to end" operation. This kind of a "loop" check is crucial for e-commerce situations.Database Access. In e-commerce applications you are either building data up or retrieving data from a database. How does that interaction perform in real world use? If you give in "correct" or "specified" input does the result produce what you expect? In e-commerce applications you are either building data up or retrieving data from a database. How does that interaction perform in real world use? If you give in "correct" or "specified" input does the result produce what you expect?Some access to information from the database may be appropriate, depending on the application, but this is typically found by other means. Navigation. Users move to and from pages, click on links, click on images (thumbnails), etc. Navigation in a Web Site is often complex and has to be quick and error free. Users move to and from pages, click on links, click on images (thumbnails), etc. Navigation in a Web Site is often complex and has to be quick and error free.Object Mode. The display you see changes dynamically; the only constants are the "objects" that make up the display. These aren't real objects in the OO sense; but they have to be treated that way. So, the quality test tools have to be able to handle URL links, forms, tables, anchors, buttons of all types in an "object like" manner so that validations are independent of representation. The display you see changes dynamically; the only constants are the "objects" that make up the display. These aren't real objects in the OO sense; but they have to be treated that way. So, the quality test tools have to be able to handle URL links, forms, tables, anchors, buttons of all types in an "object like" manner so that validations are independent of representation.Server Response. How fast the Web Site host responds influences whether a user (i.e. someone on the browser) moves on or gives up. Obviously, Internet loading affects this too, but this factor is often outside the Webmaster's control at least in terms of how the Web Site is written. Instead, it seems to be more an issue of server hardware capacity and throughput. Yet, if a Web Site becomes very popular -- this can happen overnight! -- Loading and tuning are real issues that often are imposed -- perhaps not fairly -- on the Web Master. How fast the Web Site host responds influences whether a user (i.e. someone on the browser) moves on or gives up. Obviously, Internet loading affects this too, but this factor is often outside the Webmaster's control at least in terms of how the Web Site is written. Instead, it seems to be more an issue of server hardware capacity and throughput. Yet, if a Web Site becomes very popular -- this can happen overnight! -- Loading and tuning are real issues that often are imposed -- perhaps not fairly -- on the Web Master.Interaction & Feedback. For passive, content-only sites the only real quality issue is availability. For a Web Site that interacts with the user, the big factor is how fast and how reliable that interaction is. For passive, content-only sites the only real quality issue is availability. For a Web Site that interacts with the user, the big factor is how fast and how reliable that interaction is.Concurrent Users. Do multiple users interact on a Web Site? Can they get in each others way? While Web Sites often resemble client/server structures, with multiple users at multiple locations a Web Site can be much different, and much more complex, than complex applications. Do multiple users interact on a Web Site? Can they get in each others way? While Web Sites often resemble client/server structures, with multiple users at multiple locations a Web Site can be much different, and much more complex, than complex applications.Website Test Automation Requirements Assuring Web Site quality requires conducting sets of tests, automatically and repeatable, that demonstrates required properties and behaviors. Here are some required elements of tools that aim to do this. Test Sessions. Typical elements of tests involve these characteristics: Typical elements of tests involve these characteristics:Browser Independent. Tests should be realistic, but not be dependent on a particular browser, whose biases and characteristics might mask a Web Site's problems. Tests should be realistic, but not be dependent on a particular browser, whose biases and characteristics might mask a Web Site's problems.No Buffering, Caching Local caching and buffering -- often a way to improve apparent performance -- should be disabled so that timed experiments are a true measure of the Browser-Web-Web Site-Web-Browser response time. Local caching and buffering -- often a way to improve apparent performance -- should be disabled so that timed experiments are a true measure of the Browser-Web-Web Site-Web-Browser response time.Fonts and Preferences Most browsers support a wide range of fonts and presentation preferences, and these should not affect how quality on a Web Site is assessed or assured. Most browsers support a wide range of fonts and presentation preferences, and these should not affect how quality on a Web Site is assessed or assured.Object Mode Edit fields, push buttons, radio buttons, check boxes, etc. All should be treatable in object mode, i.e. independent of the fonts and preferences. Edit fields, push buttons, radio buttons, check boxes, etc. All should be treatable in object mode, i.e. independent of the fonts and preferences.Object mode operation is essential to protect an investment in test suites and to assure that test suites continue operating when Web Site pages experience change. In other words, when buttons and form entries change location on the screen -- as they often do -- the tests should still work. However, when a button or other object is deleted, that error should be sensed! Adding objects to a page clearly implies re-making the test. Tables and Forms Even when the layout of a table or form varies in the browser's view, tests of it should continue independent of these factors. Even when the layout of a table or form varies in the browser's view, tests of it should continue independent of these factors.Frames Windows with multiple frames ought to be processed simply, i.e. as if they were multiple single-page frames. Windows with multiple frames ought to be processed simply, i.e. as if they were multiple single-page frames.Test Context. Tests need to operate from the browser level for two reasons: (1) this is where users see a Web Site, so tests based in browser operation are the most realistic; and (2) tests based in browsers can be run locally or across the Web equally well. Local execution is fine for quality control, but not for performance measurement work, where response time including Web-variable delays reflective of real-world usage is essential. Tests need to operate from the browser level for two reasons: (1) this is where users see a Web Site, so tests based in browser operation are the most realistic; and (2) tests based in browsers can be run locally or across the Web equally well. Local execution is fine for quality control, but not for performance measurement work, where response time including Web-variable delays reflective of real-world usage is essential.Website Dynamic Validation Confirming validity of what is tested is the key to assuring Web Site quality -- the most difficult challenge of all. Here are four key areas where test automation will have a significant impact. Operational Testing. Individual test steps may involve a variety of checks on individual pages in the Web Site: Individual test steps may involve a variety of checks on individual pages in the Web Site:Page Consistency, Is the entire page identical with a prior version? Are key parts of the text the same or different? Is the entire page identical with a prior version? Are key parts of the text the same or different?Table, Form Consistency, Are all of the parts of a table or form present? Correctly lay out? Can you confirm that selected texts are in the "right place". Are all of the parts of a table or form present? Correctly lay out? Can you confirm that selected texts are in the "right place".Page Relationships. Are all of the links on a page the same as they were before? Are there new or missing links? Are there any broken links? Are all of the links on a page the same as they were before? Are there new or missing links? Are there any broken links?Performance Consistency, Response Times. Is the response time for a user action the same as it was (within a range)? Is the response time for a user action the same as it was (within a range)?Test Suites. Typically you may have dozens or hundreds (or thousands?) of tests, and you may wish to run tests in a variety of modes: Typically you may have dozens or hundreds (or thousands?) of tests, and you may wish to run tests in a variety of modes:Unattended Testing. Individual and/or groups of tests should be executable singly or in parallel from one or many workstations. Individual and/or groups of tests should be executable singly or in parallel from one or many workstations.Background Testing. Tests should be executable from multiple browsers running "in the background" on an appropriately equipped workstation. Tests should be executable from multiple browsers running "in the background" on an appropriately equipped workstation.Distributed Testing. Independent parts of a test suite should be executable from separate workstations without conflict. Independent parts of a test suite should be executable from separate workstations without conflict.Performance Testing. Timing in performance tests should be resolved to the millisecond; this gives a strong basis for averaging data. Timing in performance tests should be resolved to the millisecond; this gives a strong basis for averaging data.Random Testing. There should be a capability for randomizing certain parts of tests. There should be a capability for randomizing certain parts of tests.Error Recovery. While browser failure due to user inputs is rare, test suites should have the capability of desynchronizing after an error. While browser failure due to user inputs is rare, test suites should have the capability of desynchronizing after an error.Content Validation. Apart from how a Web Site responds dynamically, the content should be checkable either exactly or approximately. Here are some ways that content validation could be accomplished: Apart from how a Web Site responds dynamically, the content should be checkable either exactly or approximately. Here are some ways that content validation could be accomplished:Structural. All of the links and anchors should match with prior "baseline" data. Images should be characterized by byte-count and/or file type or other file properties. All of the links and anchors should match with prior "baseline" data. Images should be characterized by byte-count and/or file type or other file properties.Checkpoints, Exact Reproduction, One or more text elements -- or even all text elements -- in a page should be mark-able as "required to match". One or more text elements -- or even all text elements -- in a page should be mark-able as "required to match".Gross Statistics. Page statistics (e.g. line, word, byte-count, checksum, etc.) Page statistics (e.g. line, word, byte-count, checksum, etc.)Selected Images/Fragments. The tester should have the option to rubber band sections of an image and require that the selection image match later during a subsequent rendition of it. This ought to be possible for several images or image fragments. The tester should have the option to rubber band sections of an image and require that the selection image match later during a subsequent rendition of it. This ought to be possible for several images or image fragments.Load Simulation. Load analysis needs to proceed by having a special purpose browser act like a human user. This assures that the performance checking experiment indicates true performance -- not performance on simulated but unrealistic conditions. There are many "http torture machines" that generate large numbers of http requests, but that is not necessarily the way real-world users generate requests. Load analysis needs to proceed by having a special purpose browser act like a human user. This assures that the performance checking experiment indicates true performance -- not performance on simulated but unrealistic conditions. There are many "http torture machines" that generate large numbers of http requests, but that is not necessarily the way real-world users generate requests.Sessions should be recorded live or edited from live recordings to assure faithful timing. There should be adjustable speed up and slow down ratios and intervals. Load generation should proceed from: Single Browser Sessions, One session played on a browser with one or multiple responses. Timing data should be put in a file for separate analysis. One session played on a browser with one or multiple responses. Timing data should be put in a file for separate analysis.Multiple Independent Browser Sessions, Multiple sessions played on multiple browsers with one or multiple responses. Timing data should be put in a file for separate analysis. Multivariate statistical methods may be needed for a complex but general performance model. Multiple sessions played on multiple browsers with one or multiple responses. Timing data should be put in a file for separate analysis. Multivariate statistical methods may be needed for a complex but general performance model.Summary We do Web Testing for the betterment of our Web Sites. By Web Testing we improve web sites quality and all crucial factor for a site. Tests are done to give a better service to our browsers. Author: Anuradha Panda. |
|
| Last Updated ( Thursday, 15 June 2006 ) |
