3rd Generation Frameworks : Keyword-driven

Third Generation frameworks are the most defined frameworks, and should be considered when the automation scope is relatively high, and when at least one of the automation resources has strong technical and logical skills, and a proficiency in the automation tool being used to develop the framework. The two frameworks that fit into this generation include Keyword-driven and Model-based frameworks.

Model-based PDF Print E-mail

Often called an “Intelligent framework”, Model-based frameworks go beyond creating automated tests that are executed by the tool. These frameworks are typically “given” information about the application, and the framework “creates” and executes tests in a semi-intelligent manner. Test automators describe the features of an application, typically through state models which depict the basic actions that may be performed on the application, as well as the broad expected reactions. Armed with this information the framework dynamically implements tests on the application.


Model-based Pros

* Increased coverage overtime – With a minimal amount of scripting, a lot of the application may be tested. The coverage may not necessarily be extremely high per execution, but given the random nature of the testing, the application test coverage will increase over time.
* Increased application exploration – Test automation is typically created to perform a specific set of test sequences with each test execution. Model-based test automation frameworks contain a certain degree of artificial intelligence, which allows it to perform an almost exploratory type of testing, in a way that is similar to how a manual tester might explore the application.
* Increased potential for defects discovery – Test automation is normally not meant to uncover a lot of new defects. Test automation is normally used to ensure existing functionality still works. The exploratory nature of Model-based frameworks, however, increase the chances of new defects being uncovered, given the fact that new ground is covered with each automated test run.


Model-based Challenges (Con)

* Requires higher degree of application knowledge – In order to maintain Model-based frameworks, a higher degree of application knowledge is required. This is directly related to the fact that there is less maintenance of automated tests (which is largely a technical activity), and more maintenance of application models that define application behavior.
* Required technical expertise – The technical skills required to create and maintain Model-based frameworks is relatively high. There are numerous dependencies and relationships that must be understood and maintained, as well as advanced tool components and structures.
* Increased management support – Management support is probably the most challenging with Model-based frameworks. Mainly because it is a lot simpler to calculate and communicate ROI with other frameworks. Model-based ROI calculations will more than likely be based largely on risks. Risk ROI calculations alone are often very difficult to convey to management as a justification for the time and resources necessary for creating and maintaining the structures, documentation, and personnel (both technical and non-technical).

Keyword-driven PDF Print E-mail

Often called “Table-driven”, this framework tends to be more application-independent than other frameworks. This framework processes automated tests that are developed in data tables with a vocabulary of Keywords (aka. 'Action' words) that are independent of the automated test tool used to execute them.


Screen Object Action Value Recovery Comment
LoginScreen Username Input “John”
LoginScreen Password Input “JPass”
LoginScreen Login Click
Welcome Verify_Screen Abort_Test



The keywords are associated with application-specific and application-independent functions and scripts that interpret the keyword data tables along with its application-specific data parameters. The automated scripts function as Drivers to execute the interpreted statements in the application under test.


1 Open Keyword File
2 Execute Data rows to end of file (EOF)
3 If == Input Then
4 Call Input function with Screen, Object and Value parameters
5 If == Click Then
6 Call Click function with Screen and Object parameters
7 If == Verify_Screen Then
8 Execute Verify_Screen function with Screen parameter
9 Implement Exception Handling Routine Based on Pass/Fail Status
10 End Loop


Keyword Driven frameworks rely heavily on Functional Decomposition and Data Driven framework concepts. The sample driver script above opens the Keyword file, then reads each of the rows in the file. In each row, it reads the Action statement, then calls the appropriate function. The arguments of the function are based on other fields in that row.


Keyword Driven Pros

* Increased Reusability – Reusability is even further increased with a Keyword Driven framework, because most of the functions are created in such a way to not only be reusable for multiple tests within a single application, but also for tests across multiple applications. Redundancy may therefore be decreased across all applications that an organization may be responsible for automating.
* Earlier script development – This framework increases the ability for automation to begin before the application is delivered. Using information gathered from Requirements or other documentation, keyword data tables can be created that mirror corresponding manual test procedures.
* The script is easier to read – Keyword data tables are often easier to read than regular test scripts, because the keyword data tables mirror manual test procedures. The keywords are typically verb-like phrases that make reading a keyword data table similar to reading a collection of sentences, which is easier than reading code statements that don’t mirror natural language.
* Increased standardization – The increase of reusable framework components is followed by increased standardization. The added advantage that the Keyword Framework has over Functional Decomposition, however, is that standards are by default imposed through the implementation of a Keyword framework. Provided that the framework components are created with appropriate standards, they will be invoked in the keyword data tables with every use of the framework’s keywords. Standardization helps with script maintenance, because it decreases guess work involved in figuring out what a script does and how best to fix it.
* Error handling easier to introduce – Patch work error handling solutions on a script by script basis are difficult to introduce and maintain. With reusable Keyword framework components, error handling can be introduced that reaches multiple scripts across multiple applications. This will ultimately improve the effectiveness of unattended execution of the test scripts.
* Decreases technical nature of application automation – Working with keyword data tables for everyday application automation is a lot less technical than working with code statements. Therefore individuals that are not as technical can be brought onto the team to help create automated tests.
* Greater traceability to manual test cases – Given the fact that keyword data tables so closely resemble a manual test procedure, it becomes simpler to trace actions in automated tests to actions in manual tests. In addition, there will be greater reuse of manual test procedures.


Keyword Driven Challenges (Cons)

* Required technical expertise – While the technical nature of creating automated tests within the framework is decreased, the technical skills required to create and maintain the framework itself is greatly increased from that required for Functional Decomposition frameworks. There are numerous dependencies and relationships that must be understood and maintained, as well as advanced tool components and structures.
* Intuition is less useful – In order to effectively implement a Keyword framework, reliance on intuition must be reduced, while reliance on standards must be increased. While some standards are automatically imposed, with this type of framework, many standards are not, so there’s an ongoing effort to ensure resources are aware of standards, understand them, and are able to effectively implement them. Increased documentation will probably be required to identify framework features, particularly documentation that chronicles the keywords that exist as part of the framework that may be used.
* Increased management support – Management support is a challenge for any automation effort, but it is particularly difficult with Keyword Driven frameworks. There must be strong management support, however, for the time and resources necessary for creating and maintaining the structures, documentation, and personnel (both technical and non-technical).
* Restrictive for technical staff – For technically adept resources that are tasked with day-to-day automation of a software application, Keyword frameworks may be overly restrictive. They may be perceived as an entity that “ties their hands” into automating in a “standard” way at the expense of automating in the most efficient way for a particular application, or particular feature within an application. Keyword applications typically require increased “public relations” work to sale the approach to both resources and management.

2nd Generation Frameworks: DATA DRIVEN

This is the middle ground for test automation frameworks, and may be simple, or really well defined. This framework should be considered whenever maintenance is a factor. It is important to have a strong understanding of this generation, since third-generation frameworks are also based on second-generation concepts. The two frameworks that fit into this generation are the Data-driven framework and Functional Decomposition framework.

Most Second Generation frameworks are a hybrid of both Data-driven and Functional Decomposition, but it is possible to have one without the other, so they will be discussed independently.

Functional Decomposition PDF Print E-mail

Functional Decomposition refers broadly to the process of producing modular components (user-defined functions) in such a way that automated test scripts can be constructed to achieve a testing objective largely by combining these existing components.





The modular components are often created to correspond with application functionality, but there are a lot of different types of user-defined functions that may be created. These function types may include the following:

* Utility functions – These are functions that represent fundamental framework actions or the basic functionality of a specific application under test. For example, a function that is responsible for logging into the application would be considered a utility function. These functions can be identified from the requirements or from the manual test cases, related to the groups of tests that are slated for automation.
* Navigation functions – Most applications have several main areas that are navigated to many times during testing. The paths are basic paths that can often very easily be identified at the very beginning of the automation process.
* Error handling functions – Error handling functions are functions created to deal with certain unexpected events that may occur during testing.
* Miscellaneous functions – These are functions that don’t fall under any of the other categories.

The Functional Decomposition framework may vary from having a relatively low level of complexity to high level of complexity, based on the level at which functions are created. Functions may be created for simple tasks such as menu clicks, or may be created for complex functional activities, complex error handling routines, and complex reporting mechanisms.


Functional Decomposition Script Pros

* Increased Reusability – Reusability is greatly increased with the introduction of functional decomposition, because reusable functions may be created and made available for use by multiple tests. Redundancy may be decreased for all tests within a given application, and may also be reduced across multiple applications, depending on the level of abstraction at which the function is created. With increased reusability maintainability is greatly increased.
* Script Independence – Although the Functional Decomposition Framework may use multiple external components, the tests themselves are not being reused. This allows for the introduction of reusability while still maintaining script independence.
* Earlier script development – Functional decomposition makes it possible to in some instances begin developing automated tests even before the application is delivered. Using information gathered from Requirements or other documentation, placeholder components can be created and used for developing automated tests in a top-down development approach. Once the application becomes available the components can be appropriately updated.
* The script is easier to read – Automated tests are easier to maintain when broken into small components because it is simpler to visually determine what the script is trying to accomplish.
* Increased standardization – With the increase of reusable framework components comes increased standardization. Standardization aids in script development, because it decreases much of the guess work involved in creating scripts. In addition, standardization helps with script maintenance, because it decreases guess work involved in figuring out what a script does and how best to fix it.
* Error handling is easier to introduce – Patch work error handling solutions on a script by script basis are difficult to introduce and maintain. With reusable components, error handling that reaches multiple scripts can be introduced. This will ultimately improve the effectiveness of unattended execution of the test scripts.


Functional Decomposition Script Challenges (Cons)

* Required technical expertise – As an automated test framework begins to utilize more technical solutions, that framework, by default, begs for more technically adept resources to design, develop and maintain it. This concept holds true for the functional decomposition framework.
* Intuition is less useful – In order to effectively implement an advanced framework, reliance on intuition must be reduced, while reliance on standards must be increased. While standardization is a positive by product of Functional Decomposition, it also poses a challenge in ensuring resources are aware of standards, understand them, and are able to effectively implement them. Increased documentation will probably be required to identify framework features, particularly documentation that chronicles the functions that exist as part of the framework.
* Maintenance becomes more complex – As the complexity of the framework increases, so does the amount of maintenance that is required. With Linear scripts, maintenance is confined solely to a script that breaks. While, this may result in excess maintenance for Linear scripts (because a single application change can make multiple scripts susceptible to failures), it also helps make maintenance a little less complex. With Functional Decomposition, maintenance is often required for both the framework and specific scripts. While this may reduce the amount of maintenance required, it also makes maintenance a little more complex.

Data-driven PDF Print E-mail

Frameworks built on Data-driven scripting are similar to Linear scripts, in that most of the components that are executed by a Data-driven script largely exist within the body of that script. The difference is seen in how the data is handled. The data used in Data-driven scripts is typically stored in a file that is external to the script. By parameterizing application input fields and associating the external data to its respective parameter, the run-time data will no longer be hard-coded in the script, but rather obtained from the external file. For example, below is a linear script example.


1 Input “John” into Username textbox
2 Input “JPass” into Password textbox
3 Click Login button
4 If “Welcome Screen” exists then
5 Pass the test
6 Else
7 Fail the test
8 End If



A Data-driven version of this script would store the data in an external data file (spreadsheet, database, xml file, etc.), as shown below.



NameParameter


PasswordParameter
John JPass
Sue SPass
Randy RPass
Trina TPass



The data-driven version of the Linear script shown above, may appear as follows:


1 Open Data Table
2 Input into Username textbox
3 Input into Password textbox
4 Click Login button
5 If “Welcome Screen” exists then
6 Pass the test
7 Else
8 Fail the test
9 End If
10 Close Data Table



Notice the data (John, JPass) has been removed from the script, and replaced with data table parameters.

This approach allows for the same blocks of code to be used for different data. Data-driven automation is often considered a technique as opposed to a framework, and when used as a technique, you may have several data-driven constructs within a single test script. Using the Data-driven approach as a framework does little to address many of the challenges that exist in Linear scripts, but reusability is introduced and is useful under certain circumstances.
Data-driven Script Pros

* Reusability – Data-driven scripts offer a relatively simple way to introduce reusability to automated tests. The same block of code can be reused with different data, making it possible for many test cases to be executed with minimal scripting.
* Simple to implement – It’s simple in that it requires minimal updates to existing scripts. It often involves creating parameters in place of hard coded values and placing looping structures around your code.


Data-driven Script Challenges (Cons)

* Redundancy – While Data-driven scripts allow for some degree of reusability, there is still a large degree of redundancy across multiple scripts that perform similar actions. As with Linear scripts, functionality may be duplicated in each respective script, making the scripts susceptible to excessive maintenance.
* One dimensional scripts – Little flexibility is offered in altering the way Data-driven scripts may be executed. Scripts are only readily available to be run at one level, in one location and in one way. If a test manager wants the automator to execute a subset of the total test bed based on defined priorities and risks, or execute the tests in a different environment, or in a different sequence, the automator would have to perform time consuming, run-time analysis of the scripts to make it happen. This will decrease the amount of time available for script execution and analysis.
* Script is difficult to read – With the absence of reusable components, Data-driven scripts are replete with code or automation statements. This will make the script difficult to read and therefore tedious to analyze and maintain. If a statement in the script fails, it is often difficult to determine what application functionality was affected by failure. Likewise, when it becomes necessary to make changes to the script, it is a little more tedious to determine where the changes need to be made.
* Requires higher degree of application knowledge – In order to maintain Data-driven scripts, a higher degree of application knowledge is required. This is directly related to the fact that there is little modularity. Modularity helps to make it clear what a block of application statements is responsible for doing. The loose statements in Linear Scripts make it a little less clear. So when it comes to analyzing and/or debugging a script the automator must first burn a substantial amount of time gaining a mental understanding of what a given block of statements does in order to effectively maintain the script.

1st Generation Frameworks LINEAR

1st Generation Frameworks
Framework.jpg The first generation of test automation is primarily comprised of the Linear approach to automated test development. The Linear approach typically yields a one-dimensional set of automated tests in which each automated test is treated simply as an extension of its manual counterpart. Driven mostly by the use of the Record & Playback, all components that are executed by a Linear script largely exist within the body of that script. There is little to no modularity, reusability, or any other Quality Attribute considered in the creation of Linear scripts. Linear Scripts, may be useful in environments with a very small scope.
Linear
The Linear "framework" typically yields a one-dimensional set of automated tests in which each automated test is treated simply as an extension of its manual counterpart. Driven mostly by the use of the Record & Playback, all components that are executed by a Linear script largely exist within the body of that script. There is little to no modularity, reusability, or any other Quality Attribute considered in the creation of Linear scripts. Linear Scripts, may be useful in environments with a very small scope. An example of a small linear script is shown below.
1 Input “John” into Username textbox
2 Input “JPass” into Password textbox
3 Click Login button
4 If “Welcome Screen” exists then
5 Pass the test
6 Else
7 Fail the test
8 End If

Notice that there are not calls to external modules, or external data.

Linear scripts are often driven by Record & Playback should, but in most cases, Record & Playback should not be treated as a framework or serious approach for an automation effort meant to span any real length of time. Some will agree, however, that while Record and Playback is not good as a framework, it may be useful as a technique.

Linear Script Pros
* Quick development – There is little planning or consideration of the various Quality Attributes when creating Linear scripts. Couple this information with the fact that Record and Playback is often the driving force behind Linear script development, then it’s easy to see that cost of development in time, resources and money could be relatively low.
* Smaller tool learning curve – Test automation typically requires knowledge of the automation tool features and language, as well as the knowledge of how the tool interacts with the application under test. The use of Record and Playback in the creation of Linear scripts dynamically creates code that corresponds with actions performed in the application. This code may then be studied by the automator for answers to questions about the tool’s language syntax as well as how the automated test tool’s interaction with the AUT.
* Quickly gather information about application dynamics – This advantage is closely related to the smaller tool learning curve advantage. The use of Record and Playback provides information about the Aut dynamics, such as how object properties change as activity in the application is altered.
* Script Independence – Given the fact that all the components executed by a script exist within that script, there is little apprehension about making script modifications that may unknowingly affect some other script.
* Simplifies error location – The more advanced an automation framework becomes the more complex it becomes. With complexity comes challenges related to locating the source of an error, particularly for those that are unfamiliar with the makeup of the automation framework. Given the fact that all components of a Linear script exist within that script, there is little question about where the error occurred.

Linear Script Challenges (Cons)

* Improper playback – When relying heavily on Record and Playback, recorded scripts will often not playback properly. This is due to the fact that the Record and Playback feature is not analytical. It doesn’t evaluate the application objects and behavior nor does it make decisions on how best to deal with those objects and behaviors. In an effort to fix the playback issues, a series of patchwork solutions are often introduced that won’t stand the test of time.
* Redundancy – Linear scripts don’t take advantage of reuse, therefore if more than one script performs the same or similar functionality, the functionality will be duplicated in each respective script. When the application changes in subsequent releases of the application, excessive maintenance will be the result.
* One dimensional scripts – Little flexibility is offered in altering the way Linear scripts may be executed. Scripts are only readily available to be run at one level, in one location and in one way. If a test manager wants the automator to execute a subset of the total test bed based on defined priorities and risks, or execute the tests in a different environment, or in a different sequence, the automator would have to perform time consuming, run-time analysis of the scripts to make it happen. This will decrease the amount of time available for script execution and analysis.
* Scripts are difficult to read – With the absence of reusable components, Linear scripts are replete with code or automation statements. This will make the script difficult to read and therefore tedious to analyze and maintain. If a statement in the script fails, it is often difficult to determine what application functionality was affected by failure. Likewise, when it becomes necessary to make changes to the script, it is a little more tedious to determine where the changes need to be made.
* Requires higher degree of application knowledge – In order to maintain Linear scripts, a higher degree of application knowledge is required. This is directly related to the fact that there is little modularity. Modularity helps to make it clear what a block of application statements is responsible for doing. The loose statements in Linear Scripts make it a little less clear. So when it comes to analyzing and/or debugging a script the automator must first burn a substantial amount of time gaining a mental understanding of what a given block of statements does in order to effectively maintain the script.