mirrors/eiffel-org
1
0
Fork 0
mirror of https://github.com/EiffelSoftware/eiffel-org.git synced 2025-12-08 15:52:26 +01:00
Code Issues Packages Projects Releases Wiki Activity

Author:halw

Browse Source 
Date:2010-01-19T20:15:45.000000Z


git-svn-id: https://svn.eiffel.com/eiffel-org/trunk@398 abb3cda0-5349-4a8f-a601-0c33ac3a8c38
This commit is contained in:
halw
2010-01-19 20:17:40 +00:00
parent 563c5218a2
commit db4202f903
5 changed files with 168 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files

91
documentation/current/eiffelstudio/eiffelstudio-guided-tour/using-autotest/create-manual-test.wiki
View File

@@ -1,7 +1,6 @@
[[Property:title|Create a manual test]]
[[Property:weight|2]]
[[Property:uuid|e78f25e3-ed3a-f8fa-e71d-28a4dda1825f]]
==A system to test==
For developing our manual test, let's use a simple system that contains a class modeling bank accounts. Here are two classes that will make up our system. The first, <code>APPLICATION</code> will be the root class of our system. <code>APPLICATION</code> really only serves to declare an attribute of type <code>BANK_ACCOUNT</code>, which is the class we will write a test against. <code>APPLICATION</code> looks like this:
@@ -79,18 +78,18 @@ You shouldn't let it worry you if you've noticed that the class <code>BANK_ACCOU
If you want to work along with this tutorial, you should be able to copy the text of each these classes from this page and paste it into the EiffelStudio editor pane. Build a system using these two classes, and <code>{APPLICATION}.make</code> as the root.
{{note|If you are using EiffelStudio version 6.3, there two things you will need to do to prepare your system for use with the Eiffel Testing Framework. Both of these are done from the [[EiffelStudio: Project settings window]].<br/> 1) Set your project to be a console application in the [[Advanced options]].<br/>2) Set a value of <code>False</code> for the <code>Recursive</code> attribute of your project cluster in [[Group options]].}}
{{note|If you are using EiffelStudio version 6.3, there two things you will need to do to prepare your system for use with AutoTest. Both of these are done from the [[EiffelStudio: Project settings window]].<br/> 1) Set your project to be a console application in the [[Advanced options]].<br/>2) Set a value of <code>False</code> for the <code>Recursive</code> attribute of your project cluster in [[Group options]].}}
==Getting to the Eiffel Testing Framework interface==
==Getting to the AutoTest interface==
If the Eiffel Testing Framework interface is not on a tab next to Clusters, Features, and Favorites, you can invoke it by following the menu path:
If the AutoTest interface is not on a tab next to Clusters, Features, and Favorites, you can invoke it by following the menu path:
<code lang=text>
View --> Tools --> Testing Tool
View --> Tools --> AutoTest
</code>
Depending upon your version and platform, the Eiffel Testing Framework interface should look about like this:
Depending upon your version and platform, the AutoTest interface should look about like this:
[[Image:empty testing tool 01]]
[[Image:AutoTest empty tool 01]]
==Creating a new test==
@@ -102,9 +101,9 @@ This will launch the New Eiffel Test Wizard which guides you through the test cr
If this is the first time you've used the testing tool for this project, it is likely that you will be presented with a dialog box asking if you want to add the testing library classes to your project and recompile:
[[Image:add testing libraries dialog]]
[[Image:AutoTest add testing libraries dialog]]
You want EiffelStudio to do this before launching the wizard so, click "Yes". In a moment your system will have recompiled with the testing library classes available. Remember that you won't need to interact much with the testing classes, but the Eiffel Testing Framework uses them, so they need to be available. As long as the testing classes stay available, you should not see this dialog again for the current project.
You want EiffelStudio to do this before launching the wizard so, click "Yes". In a moment your system will have recompiled with the testing library classes available. Remember that you won't need to interact much with the testing classes, but AutoTest uses them, so they need to be available. As long as the testing classes stay available, you should not see this dialog again for the current project.
==The New Eiffel Test Wizard==
@@ -112,36 +111,36 @@ You want EiffelStudio to do this before launching the wizard so, click "Yes". In
After the compile completes, then the first pane of the New Eiffel Test Wizard appears. It should look like this:
[[Image:New test wizard screen 01 01]]
[[Image:AutoTest new test wizard 01 01]]
As you can see, there is a radio button for each of the three types of tests that were discussed in [[Testing: Background and basics]]. The buttons for '''Manual''' and '''Synthesized''' tests are enabled. The button for '''Extracted''' tests is not sensitive because the extraction of tests is only valid during the time in which a system is actually running. That's okay, we were going to choose '''Manual''' anyway.
As you can see, there is a radio button for each of the three types of tests that were discussed in [[Testing: Background and basics]]. The buttons for '''Manual''' and '''Generated''' tests are enabled. The button for '''Extracted''' tests is not sensitive because the extraction of tests is only valid during the time in which a system is actually running. That's okay, we were going to choose '''Manual''' anyway.
If we select '''Manual''' and click '''Next''', then second pane of the wizard appears:
[[Image:New test wizard screen 02M 01]]
[[Image:AutoTest new test wizard 02M 01]]
We will use this wizard pane to name our test class and let the Eiffel Testing Framework know where we want the test class to reside. You can give a test class any name you wish, as long as it doesn't conflict with another class name in your system. If you try to type in a class name that already exists, the wizard will let you know right away by changing the text color to red. There is a convention that has arisen around test class names. If possible make the test class name the name of the target class, prefixed with <code>TEST_</code>. So in our case, we want to build a test against a feature of the <code>BANK_ACCOUNT</code> class, so we will name our test class <code>TEST_BANK_ACCOUNT</code>.
We will use this wizard pane to name our test class and let AutoTest know where we want the test class to reside. You can give a test class any name you wish, as long as it doesn't conflict with another class name in your system. If you try to type in a class name that already exists, the wizard will let you know right away by changing the text color to red. There is a convention that has arisen around test class names. If possible make the test class name the name of the target class, prefixed with <code>TEST_</code>. So in our case, we want to build a test against a feature of the <code>BANK_ACCOUNT</code> class, so we will name our test class <code>TEST_BANK_ACCOUNT</code>.
Now, for the question of where the tests should be kept. Again, you can keep them anywhere you wish, but there are some advantages to keeping the test classes in a '''test cluster''' separate from your target classes. For example, it will be easier for you to deliver your application or library classes if the testing classes are mixed it. A '''test cluster''' is just a cluster of classes that EiffelStudio and the Eiffel Testing Framework expect to contain test classes. So we want to create a new testing cluster as a subcluster of the cluster in which the classes <code>APPLICATION</code> and <code>BANK_ACCOUNT</code> reside.
Now, for the question of where the tests should be kept. Again, you can keep them anywhere you wish, but there are some advantages to keeping the test classes in a '''test cluster''' separate from your target classes. For example, it will be easier for you to deliver your application or library classes if the testing classes are mixed it. A '''test cluster''' is just a cluster of classes that EiffelStudio and AutoTest expect to contain test classes. So we want to create a new testing cluster as a subcluster of the cluster in which the classes <code>APPLICATION</code> and <code>BANK_ACCOUNT</code> reside.
Notice the '''New cluster''' button ( [[Image:16x16--new-cluster-icon]] ) on the wizard pane above. We click that button to add a new test cluster. The '''Add Cluster''' dialog box appears:
[[Image:New test wizard add cluster 01]]
[[Image:AutoTest new test wizard Add Cluster 01]]
As with test class names, there is an emerging convention for the name of a test cluster: <code>tests</code> . So we will name our test cluster <code>tests</code> and make it a subcluster to our root cluster <code>bank_account</code>. Notice that there is a '''test cluster''' check box on the dialog. It is checked and disabled, so at this point in the wizard you would always create a test cluster. Once the test cluster is created, we're back to the second wizard pane which now looks like this:
As with test class names, there is an emerging convention for the name of a test cluster: <code>tests</code> . So we will name our test cluster <code>tests</code> and make it a subcluster to our root cluster <code>accounts</code>. Notice that there is a '''test cluster''' check box on the dialog. It is checked and disabled, so at this point in the wizard you would always create a test cluster. Once the test cluster is created, we're back to the second wizard pane which now looks like this:
[[Image:New test wizard screen 02M 02]]
[[Image:AutoTest new test wizard 02M 02]]
We're ready to click '''Next''', but before we do, let's look at the check boxes on this wizard pane. The two check boxes labeled '''Redefine `on_prepare'''' and '''Redefine `on_clean'''' have to do with the way that tests are run.
The Eiffel Testing Framework runs each test as a three step process:
AutoTest runs each test as a three step process:
# Preparation
# Execution
# Clean up
@@ -149,13 +148,13 @@ The Eiffel Testing Framework runs each test as a three step process:
There are features in class <code>EQA_TEST_SET</code> named <code>prepare</code> and <code>clean</code> which accomplish steps 1 and 3 above. These features are <code>frozen</code>, therefore you cannot redefine them in a test class (i.e., a descendant of <code>EQA_TEST_SET</code>) However the class does provide features that can be redefined so that you can include custom behavior before and/or after the execution of a test. These features are <code>on_prepare</code> and <code>on_clean</code>. So if you check one of these boxes, then the test class that is built for you will include a redefined feature ready for you to implement. In this simple example, we'll leave both boxes unchecked.
{{note|The check box labeled '''System level test''' is displayed here as not sensitive. This box is reserved for future system level testing capability in the Eiffel Testing Framework, so for versions including 6.4, you can ignore it. }}
{{note|The check box labeled '''System level test''' is displayed here as not sensitive. This box is reserved for future system level testing capability in AutoTest, so for versions including 6.5, you can ignore it. }}
After clicking '''Next''', we see the third pane of the wizard. In this pane, we will describe our manual test routine. It looks like this:
[[Image:New test wizard screen 03M 01]]
[[Image:AutoTest new test wizard 03M 01]]
Here we will name our test. Let's say that we plan to write this test against the feature <code>{BANK_ACCOUNT}.deposit</code>. Like the naming convention for test classes, we'll give this test the name <code>test_deposit_01</code>. The prefix <code>test_</code> before the feature name it will test, and the suffix <code>_01</code> so that we have a framework for adding more tests against <code>deposit</code>. Again, you can choose any naming scheme that makes sense to you. You may want to try to describe the test in its name. For example, <code>test_deposit_very_large_amount</code>.
@@ -163,9 +162,9 @@ Here we will name our test. Let's say that we plan to write this test against th
==About Tags==
The other thing that can be done on this pane is to associate our test with any Eiffel Testing Framework '''tags''' that we feel are appropriate.
The other thing that can be done on this pane is to associate our test with any AutoTest '''tags''' that we feel are appropriate.
'''Tags''' are simply names or otherwise meaningful strings of characters that are arranged hierarchically and can be associated with a test to help manage, maintain, execute, and monitor its results. Any one test can support many tags. It is quite likely that during the development process, your system may eventually accumulate a great number of tests. And you may want only to execute some selected portion of those tests at any particular time. '''Tags''' allow you do that with the help of the Eiffel Testing Framework.
'''Tags''' are simply names or otherwise meaningful strings of characters that are arranged hierarchically and can be associated with a test to help manage, maintain, execute, and monitor its results. Any one test can support many tags. It is quite likely that during the development process, your system may eventually accumulate a great number of tests. And you may want only to execute some selected portion of those tests at any particular time. '''Tags''' allow you do that with the help of AutoTest.
One of the most common types of tags specifies what class and feature a test covers. In our example, we will write our test against the <code>deposit</code> procedure of the class <code>BANK_ACCOUNT</code>. As you will see in a moment, the tag that will express this is:
<code>
@@ -176,8 +175,14 @@ When we look at a tag in this notation, each hierarchical level is delimited by
covers/{BANK_ACCOUNT}.deposit
covers/{BANK_ACCOUNT}.withdraw
</code>
So when every you asked to view or run all the tests that <code>covers</code> either <code>deposit</code> or <code>withdraw</code>, this test would show up in that set.
The "covers" tags, as you will see, can be generated by the Eiffel Testing Framework when you create a test. But tags can be completely arbitrary, too. So, for example if you were building software that you expected to run on multiple platforms, in the test suite, you might have a test with the following tags:
So when ever you ask to view or run all the tests that <code>covers</code> either <code>deposit</code> or <code>withdraw</code>, this test would show up in that set.
The "covers" tags, as you will see, can be generated by AutoTest's New Eiffel Test Wizard when you create a new test. But you could enter the tag manually, as well. For example if you had written a high level test that exercised all or most of the functionality of the class <code>BANK_ACCOUNT</code>, you could manually add a tag that expresses that, i.e., a "covers" tag for <code>BANK_ACCOUNT</code> that does not specify a particular routine:
<code>
covers/{BANK_ACCOUNT}
</code>
Tags can be completely arbitrary, too. So, for example if you were building software that you expected to run on multiple platforms, in the test suite, you might have a test with the following tags:
<code>
platform/os/linux
platform/architecture/i386
@@ -185,7 +190,7 @@ platform/architecture/i386
So this test would be specifically for Linux running on Intel architecture. When you were testing on that platform combination, you could select the appropriate tests to run using tags.
==Associating tags with a new test==
===Associating tags with a new test===
Now that we've named our new test, let's associate a tag with it that indicates that it <code>covers</code> the <code>deposit</code> procedure of class <code>BANK_ACCOUNT</code>. Looking again at the New Eiffel test wizard pane, you will see that there are three boxes under the label '''Tags for new tests'''. The first is just a display of the list of tags that you have added to the new test. The next box down allows you to add an arbitrary tag sequence like:
<code>
@@ -194,14 +199,43 @@ platform/os/linux
And the third allows you to add certain commonly used or predefined tag types. This is the box we'll use. So first we will select '''Add class/feature under test tag''', then click the '''Add''' button to the right of the box. This will cause the appearance of a dialog that allows us to pick a target class and routine from our system. So we navigate to <code>{BANK_ACCOUNT}.deposit</code>. The dialog will look like this:
[[Image: New test wizard screen 03M dialog 01]]
[[Image: AutoTest new test wizard 03M dialog 01]]
Now we click '''OK'''. The dialog disappears and the <code>covers</code> tag:
<code>
covers/{BANK_ACCOUNT}.deposit
</code>
is now visible in the list of tags for the new test we are creating. So, next click '''Create''' and the wizard pane disappears and the Eiffel Testing Framework will create our test class and display it in the edit window.
is now visible in the list of tags for the new test we are creating. So, next click '''Create''' and the wizard pane disappears and AutoTest will create our test class and display it in the edit window.
===Other predefined tags===
In addition to '''Add class/feature under test tag''', choices for other predefined tags are shown in the drop-down box. For example, the choices for '''execution''' tags are '''Run test in private evaluator''' and '''Run test serially'''.
Selecting '''Run test in private evaluator''' will insert the tag:
<code>
execution/isolated
</code>
When tests are executed, they do so within the context of '''evaluator processes'''. Normally, evaluator processes are reused for multiple test executions. But if you select '''Run in private evaluator''', the tag added to your test guarantees that this test will be run in a fresh evaluator process, that terminates when the test completes. This can be helpful, for example, when you don't want your test to enter or leave the evaluator process with the effects of "once" routines or any other action that might effect the efficacy of other tests. For example, if your test executes external routines which might have a damaging effect on memory, you should run the test in a private evaluator.
If you select '''Run test serially''', the following tag will be inserted:
<code>
execution/serial
</code>
Tests tagged with this tag will not run concurrently with any other similarly tagged test is running.
You can extend the serial execution tag with arbitrary terms that will differentiate groups of tagged tests. For example, if some of your tests are tagged like this:
<code>
execution/serial/group_1
</code>
and some are tagged:
<code>
execution/serial/group_2
</code>
then AutoTest will not run any <code>group_1</code> tagged test concurrently with any other <code>group_1</code> test, and likewise for tests tagged <code>group_2</code>.
==Writing a test==
@@ -254,9 +288,8 @@ So, <code>test_deposit_01</code> now looks like this:
end
</code>
Now we have created and written an Eiffel Testing Framework manual test.
Now we have created and written a manual test using AutoTest.
Next let's see what it takes to execute a test.

12
documentation/current/eiffelstudio/eiffelstudio-guided-tour/using-autotest/index.wiki
View File

@@ -1,18 +1,18 @@
[[Property:title|Using the Eiffel Testing Framework]]
[[Property:link_title|Using AutoTest]]
[[Property:title|Using AutoTest]]
[[Property:weight|-1]]
[[Property:uuid|6b900a65-85c6-9cd6-ef57-ccd4b8decbef]]
==Introduction==
The Eiffel Testing Framework is a tool that helps you to create, manage, and run tests against your software. The Eiffel Testing Framework is accessible directly as a part of EiffelStudio, but works to a large extent behind the scenes so that it doesn't get in the way of your development activities. In other words, even though you may be accumulating a substantial collection of test software along with your project software, you can still run and deliver your project software without going to a lot of trouble to separate the two. Tests managed by the Eiffel Testing Framework stay handy and can be run any time to help make sure everything always stands up to the scrutiny of testing.
AutoTest is a tool that helps you to create, manage, and run tests against your software. AutoTest is accessible directly as a part of EiffelStudio, but works to a large extent behind the scenes so that it doesn't get in the way of your development activities. In other words, even though you may be accumulating a substantial collection of test software along with your project software, you can still run and deliver your project software without going to a lot of trouble to separate the two. Tests managed by AutoTest stay handy and can be run any time to help make sure everything always stands up to the scrutiny of testing.
This tutorial will guide you through the use of the Eiffel Testing Framework. A [[The Eiffel Testing Framework|reference section]] for the Framework is also available.
This tutorial will guide you through the use of AutoTest. A [[AutoTest|reference section]] for AutoTest is also available.
{{note|It is recommended, at least on your first viewing of this tutorial, that you take the sections in the order in which they are presented. There are three different types of tests supported by the Eiffel Testing Framework. Each type of test is discussed on its own page. But to avoid repetition, the pages for the second and third types of tests omit some of the detail in the first and assume a familiarity with the example. }}
{{note|It is recommended, at least on your first viewing of this tutorial, that you take the sections in the order in which they are presented. There are three different types of tests supported by AutoTest. Each type of test is discussed on its own page. But to avoid repetition, the pages for the second and third types of tests omit some of the detail in the first and assume a familiarity with the example. }}
{{SeeAlso|<br />[[The Eiffel Testing Framework]] reference }}
{{SeeAlso|<br />[[AutoTest]] reference }}

35
documentation/current/eiffelstudio/eiffelstudio-guided-tour/using-autotest/testing-background-and-basics.wiki
View File

@@ -1,7 +1,6 @@
[[Property:title|Testing: Background and basics]]
[[Property:weight|0]]
[[Property:uuid|12c2a2d4-9bf2-ba73-6647-cb9900666de1]]
==Background and motivation for testing tools==
Developers test software in the hope that the testing process will expose faults in the software they've developed. Most developers also realize that no amount of testing will ever prove software to be bug free. So while testing is a virtuous activity that we dare not neglect, we are wise to temper our expectation of the practical value of testing.
@@ -26,13 +25,13 @@ A term commonly used in software testing is "oracle". Tests are generally looked
The presence of preconditions provides another advantage. Preconditions make it possible to automate testing in ways unavailable in other environments. Because of preconditions, we already have information about the limits of valid inputs to routines. So it's possible to generate a call to a routine we want to test automatically and with a context that meets the routine's precondition.
==The Eiffel Testing Framework==
==AutoTest==
The Eiffel Testing Framework attempts to capitalize on the testing advantages inherent in Eiffel due to Design by Contract. The Eiffel Testing Framework consists of an interactive interface, and a library of classes which support testing activity.
AutoTest attempts to capitalize on the testing advantages inherent in Eiffel due to Design by Contract. AutoTest consists of an interactive interface, and a library of classes which support testing activity.
The testing support classes are distributed with EiffelStudio and exist in the ''testing'' subfolder of the ''libraries'' folder. With the exception of one class which we will discuss soon, the classes in "testing" are not intended to be used directly by developers. They exist to support the functionality of the Eiffel Testing Framework.
The testing support classes are distributed with EiffelStudio and exist in the ''testing'' subfolder of the ''libraries'' folder. With the exception of one class which we will discuss soon, the classes in "testing" are not intended to be used directly by developers. They exist to support the functionality of AutoTest.
The interface for the Eiffel Testing Framework is accessible through the EiffelStudio development environment. You may find it already resident as a tab in the right hand pane next to Clusters, Features, and Favorites. If it's not there, then you can bring it up by following the menu path:
The interface for AutoTest is accessible through the EiffelStudio development environment. You may find it already resident as a tab in the right hand pane next to Clusters, Features, and Favorites. If it's not there, then you can bring it up by following the menu path:
<code lang=text>
View --> Tools --> Testing Tool </code>
@@ -40,7 +39,7 @@ View --> Tools --> Testing Tool </code>
==Test classes and tests==
The Eiffel Testing Framework interface helps you to create and execute tests on the software you develop. The interface contains a wizard called the '''New Eiffel Test Wizard''' which helps you create or generate the types of tests you need. We'll learn more about the interface and the wizard as we go along. But first, let's look at what constitutes an Eiffel Testing Framework ''test''. For the Eiffel Testing Framework, we define the term ''test'' in the context of some other testing terminology:
The AutoTest interface helps you to create and execute tests on the software you develop. The interface contains a wizard called the '''New Eiffel Test Wizard''' which helps you create or generate the types of tests you need. We'll learn more about the interface and the wizard as we go along. But first, let's look at what constitutes a ''test''. For AutoTest, we define the term ''test'' in the context of some other testing terminology:
{{definition|Test class|An effective class that inherits from the class EQA_TEST_SET. }}
@@ -55,17 +54,17 @@ The Eiffel Testing Framework interface helps you to create and execute tests on
{{definition|Test suite|A set of test classes (and by implication the tests contained therein) which is designed to test some particular software system or library. }}
Whenever you use the Eiffel Testing Framework, it will find your test classes, those classes that inherit from EQA_TEST_SET. When you run tests, it will execute all the tests in those classes, or a subset of tests that you choose. So, you have probably figured out that the one class from the testing library that you may need to know a little about is EQA_TEST_SET. But you don't have to know very much, because the Eiffel Testing Framework can help you construct your test classes.
Whenever you use AutoTest, it will find your test classes, those classes that inherit from EQA_TEST_SET. When you run tests, it will execute all the tests in those classes, or a subset of tests that you choose. So, you have probably figured out that the one class from the testing library that you may need to know a little about is EQA_TEST_SET. But you don't have to know very much, because AutoTest can help you construct your test classes.
==Types of tests==
There are three different types of tests supported by the Eiffel Testing Framework:
There are three different types of tests supported by AutoTest:
* Manual tests
* Extracted tests
* Synthesized tests
* Generated tests
Each test of any of these types ultimately is a feature of class that inherits from EQA_TEST_SET. Ordinarily, though, the three types of tests won't be mixed in a test class. That is, any one particular test class will contain only one type of test. But from the point of view of the Eiffel Testing Framework, all types of tests are managed and run the same way. We will discuss these types of tests in more detail later, but for right now, let's just establish some definitions.
Each test of any of these types ultimately is a feature of class that inherits from EQA_TEST_SET. Ordinarily, though, the three types of tests won't be mixed in a test class. That is, any one particular test class will contain only one type of test. But from the point of view of AutoTest, all types of tests are managed and run the same way. We will discuss these types of tests in more detail later, but for right now, let's just establish some definitions.
{{definition|Manual test|A test manually coded within a test class. }}
@@ -82,12 +81,12 @@ A manual test is "manual" in the sense that you code the essential procedural pa
Extracted tests are convenient because they allow you to accumulate tests that are based on actual failures of your software (good for the software, not so good for your ego!). Once these tests are in your suite of tests, they are available from then on.
{{definition|Synthesized test|A test that is the product of generating and running a series of randomly generated invocations of target routines. }}
{{definition|Generated test|A test that is the product of generating and running a series of randomly generated invocations of target routines. }}
The process of creating synthesized tests is sometimes known in the community as creating via ''AutoTest''. The randomly generated calls to target routines which were created and run are discarded at the completion of the creation. But from the results of these calls, a set of permanent tests is distilled. These are the synthesized tests.
The process of creating generated tests is sometimes known in the community as creating via ''AutoTest''. The randomly generated calls to target routines which were created and run are discarded at the completion of the creation. But from the results of these calls, a set of permanent tests is distilled. These are the generated tests.
Synthesized tests are made possible by Design by Contract. Hopefully, you remember that one thing that DbC gives us is the handy ability to assign blame when something goes wrong. When a test makes a call to a routine we want to test, if a contract violation occurs, it may be the fault of the called routine or it may be the fault of the caller ... and that depends upon what type of contract violation has occurred. The contract violations that are interesting to the Eiffel Testing Framework in the process of synthesizing tests are only those in which the called routine is at fault. That is, postcondition and invariant violations. The Eiffel Testing Framework will then create a synthesized test for every ''unique'' failure in which the called routine being tested was to blame.
Generated tests are made possible by Design by Contract. Hopefully, you remember that one thing that DbC gives us is the handy ability to assign blame when something goes wrong. When a test makes a call to a routine we want to test, if a contract violation occurs, it may be the fault of the called routine or it may be the fault of the caller ... and that depends upon what type of contract violation has occurred. The contract violations that are interesting to AutoTest in the process of synthesizing tests are only those in which the called routine is at fault. That is, postcondition and invariant violations. AutoTest will then create a generated test for every ''unique'' failure in which the called routine being tested was to blame.
@@ -105,7 +104,7 @@ In its simplest form, a test is a routine that issues a call to some routine you
So the tests and the test classes are in the realm of testing and are used to test the target routines in target classes which are the real product of your software development project.
The Eiffel Testing Framework will manage and run the tests in any test class whether or not they actually test any target routines. Even though the test shown below doesn't test anything, it still qualifies as a test. Naturally, it would seem silly to keep a test around that doesn't test anything, but the important thing to understand is that the Eiffel Testing Framework will work with anything that matches the definitions of test and test class above. That is, once tests are created, the Eiffel Testing Framework doesn't really have a stake in what you are trying to test.
AutoTest will manage and run the tests in any test class whether or not they actually test any target routines. Even though the test shown below doesn't test anything, it still qualifies as a test. Naturally, it would seem silly to keep a test around that doesn't test anything, but the important thing to understand is that AutoTest will work with anything that matches the definitions of test and test class above. That is, once tests are created, AutoTest doesn't really have a stake in what you are trying to test.
@@ -137,12 +136,12 @@ feature -- Test routines
end</code>
This test class was created by the Eiffel Testing Framework's New Eiffel Test Wizard. It is about as simple a test class as there can be. Its only value is to illustrate the basic form of Eiffel Testing Framework tests. So, let's look at that form.
This test class was created by AutoTest's New Eiffel Test Wizard. It is about as simple a test class as there can be. Its only value is to illustrate the basic form of AutoTest tests. So, let's look at that form.
It is clear that <code>MY_TEST_CLASS</code> is an effective class that inherits from <code>EQA_TEST_SET</code>, so that makes it fit the definition of a test class. And, it's also clear the <code>my_test</code> is a feature of <code>MY_TEST_CLASS</code>, specifically a procedure, exported to <code>ANY</code>, requiring no arguments. That qualifies <code>my_test</code> as a test. If <code>MY_TEST_CLASS</code> is located in a test cluster of your project, then the Eiffel Testing Framework will find it and be able to run it whenever you request.
It is clear that <code>MY_TEST_CLASS</code> is an effective class that inherits from <code>EQA_TEST_SET</code>, so that makes it fit the definition of a test class. And, it's also clear the <code>my_test</code> is a feature of <code>MY_TEST_CLASS</code>, specifically a procedure, exported to <code>ANY</code>, requiring no arguments. That qualifies <code>my_test</code> as a test. If <code>MY_TEST_CLASS</code> is located in a test cluster of your project, then AutoTest will find it and be able to run it whenever you request.
This test would always fail because of the <code>assert</code> that the wizard put in the implementation. So if you asked the Eiffel Testing Framework to run your tests, it would tell you that <code>my_test</code> was a failed test, for the reason: "not_implemented". The <code>assert</code> is not a necessary part of a test. The wizard puts it there to remind you that the test has not been implemented. If you removed the <code>assert</code> line from the test, then the test would always succeed, which would be nice, but it would be succeeding at testing nothing! We'll see more later about what it means for tests to succeed and fail.
This test would always fail because of the <code>assert</code> that the wizard put in the implementation. So if you asked AutoTest to run your tests, it would tell you that <code>my_test</code> was a failed test, for the reason: "not_implemented". The <code>assert</code> is not a necessary part of a test. The wizard puts it there to remind you that the test has not been implemented. If you removed the <code>assert</code> line from the test, then the test would always succeed, which would be nice, but it would be succeeding at testing nothing! We'll see more later about what it means for tests to succeed and fail.
But first let's get some exposure to the Eiffel Testing Framework interface, by building a manual test for a routine in a simple class.
But first let's get some exposure to the AutoTest interface, by building a manual test for a routine in a simple class.

99
documentation/current/eiffelstudio/eiffelstudio-reference/autotest/autotest-interface.wiki
View File

@@ -1,19 +1,14 @@
[[Property:title|The Eiffel Testing Framework Interface]]
[[Property:link_title|The AutoTest Interface]]
[[Property:title|The AutoTest Interface]]
[[Property:weight|0]]
[[Property:uuid|6eec11df-9ea0-6834-d41b-a0c23b87c485]]
[[Image:AutoTest interface annotated 01]]
[[Image:Testing Framework interface annotated 01]]
The Eiffel Testing Framework interface consists of four primary components:
The AutoTest interface consists of four primary components:
# The toolbar
# The view filtering controls
# The tests view
# The tabbed status and results reporting pane
# The tests pane
# The test filtering box
# The tabbed creation and results reporting pane
==Toolbar==
@@ -21,65 +16,99 @@ The toolbar allows you to create new tests using the '''[[#The New Eiffel test w
The '''Create new tests''' command ( [[Image:Create new tests]] ) invokes the '''New Eiffel test wizard'''.
The '''Run all tests in background''' command ( [[Image:debug-run-icon]] ) will, by default, run all tests. But if only certain tests are selected in the '''Tests''' list, then '''Run all tests in background''' will run only those tests. The '''Run all tests in background''' allows other options for running tests through its drop-down icon ( [[Image:toolbar-dropdown-icon]] ).
The '''Run filtered tests''' command ( [[Image:debug-run-icon]] ) will, by default, run all tests that are visible in the Tests pane. Whether or not a test is visible depends upon any test [[#Filtering|filtering]] currently in effect. The '''Run filtered tests''' allows other options for running tests through its drop-down icon ( [[Image:toolbar-dropdown-icon]] ).
[[Image:Testing Framework run all drop down]]
[[Image:AutoTest run all drop down]]
The '''Debug all tests in EiffelStudio''' command ( [[Image:Testing Framework debug all icon]] ) functions in a similar manner to '''Run all tests in background''' with the exception that each test is paused at its starting point in the EiffelStudio debugger. '''Debug all tests in EiffelStudio''' has a drop-down icon providing the same options as '''Run all tests in background'''.
The '''Debug filtered tests''' command ( [[Image:AutoTest debug all icon]] ) functions in a similar manner to '''Run filtered tests''' with the exception that each test is paused at its starting point in the EiffelStudio debugger. '''Debug filtered tests''' has a drop-down icon providing the same options as '''Run filtered tests'''.
[[Image:Testing Framework debug all drop down]]
[[Image:AutoTest debug all drop down]]
The '''Stop all execution''' command ( [[Image:debug-stop-icon]] ) will halt all tests which were started by the '''Run all tests in background''' or '''Debug all tests in EiffelStudio''' commands.
==View Filtering==
{{note|The development of this documentation section is currently on hold pending changes to the interface expected in an upcoming release. }}
The '''Stop all test related tasks''' command ( [[Image:debug-stop-icon]] ) will halt all tests which were started by the '''Run filtered tests''' or '''Debug filtered tests''' commands.
==Tests==
The Tests pane presents tests in one or more expandable tree views. The structures of the trees depend upon the criteria that you have specified for organizing the view. For example, if you look at tests based on the test classes in which they exist, the structure of the tree view(s) will reflect the clusters in which the test classes exist.
The Tests pane presents tests in one or more expandable tree views. The structures of the trees depend upon the [[#Filtering|filter]] criteria that you have specified for organizing the view. For example, if you look at tests based on the test classes in which they exist (tag root "''class''"), the structure of the tree view(s) will reflect the clusters in which the test classes exist.
[[Image:Testing Framework interface test view - class]]
[[Image:AutoTest interface test view class]]
Likewise, if you look at tests based on their target classes, the structure of the display will be based on the target classes.
Likewise, if you look at tests based on their target classes (tag root "''covers''"), the structure of the display will be based on the target classes.
[[Image:Testing Framework interface test view - covers]]
[[Image:AutoTest interface test view covers]]
Tests and test classes are pickable in the Tests pane.
==Status and results==
==Filtering==
This pane supports a tabbed display that provides information about the execution and results of tests.
The '''Filter''' box allows you to limit which tests are visible in the Tests pane (and, by consequence, the tests run when '''Run filtered tests''' is clicked) by entering filter text in the box and pressing the Enter key. Filter text can be a string of characters that is contained in a test class or test routine name, or it can be a [[Create a manual test#About Tags|tag]] or a portion of a tag hierarchy.
After execution or debugging, the tests executed and their status are shown in the '''Execution''' tab or the '''Debugging''' tab.
In fact, filter text fully supports [http://www.regular-expressions.info/ regular expressions], so you can craft fine-grained filters.
Clicking the '''Clear filter''' icon ( [[Image:general-reset-icon]] ) to the right of the filter box will clear any filter contents and enable the viewing of all tests. The Tests pane is depicted below after clicking the '''Clear filter''' icon and expanding some of the items:
[[Image:Testing Framework interface results - Execution]]
[[Image:AutoTest tests pane null filter]]
Double clicking a test in the '''Tests''' pane will switch the '''Status and results''' pane to the '''Results''' tab. This tab shows an expandable history of the executions and results of that particular test.
The '''Filter''' box has a drop down icon on its right end. By default this icon provides views of the tests in the Tests pane based on tag criteria.
[[Image:Testing Framework interface results - Results]]
[[Image:AutoTest filter drop down]]
For failing executions, the failure code is shown, along with link to display the execution trace.
[[Image:Testing Framework interface results - trace]]
The criteria are associated with the roots of their respective tag trees. So for example, if '''Results''' is chosen the Tests pane shows tests grouped by their latest results.
[[Image:AutoTest filter result]]
{{note|The '''result''' tag hierarchy is, as you may have guessed, virtual and dynamic. A test falling into the '''fail''' subtree may later become part of the '''pass''' subtree without any change to the '''"testing:"''' notes in the test routine code, where tags are established. But for filtering purposes '''result''' tags work like other tags. }}
If the '''fail''' subtree name were appended to the '''result''' filter, only the failed tests would be visible in the Tests pane:
[[Image:AutoTest filter result fail]]
If the filter text is a character string that occurs in a test name, test class name, or a tag path, tests with any match will be visible:
[[Image:AutoTest filter withdraw]]
==Results==
This pane supports a tabbed display that provides information about the creation and execution of tests.
After creation of one or more tests, the creation is logged under the '''Creation''' tab.
After execution, the tests executed and the status of those tests are shown in the '''Execution''' tab.
[[Image:AutoTest interface results execution]]
After executing a set of tests, the Testing pane of the Outputs tool shows a chronology of the executions and the results of the tests executed.
[[Image:AutoTest testing pane execution results]]
You can view more detail about a test's execution by clicking the '''Information''' icon ( [[Image:AutoTest information icon]] ) to the right of the test's entry in '''Results''' when the '''Execution''' tab is selected.
[[Image:AutoTest test results details]]

42
documentation/current/eiffelstudio/eiffelstudio-reference/autotest/eiffel-test-wizard.wiki
View File

@@ -2,13 +2,11 @@
[[Property:link_title|The Eiffel Test Wizard]]
[[Property:weight|1]]
[[Property:uuid|305a8288-cb6a-df2a-1515-e23138e21566]]
==Introduction==
The '''New Eiffel test wizard''' creates new tests based on information entered by a developer on a series of wizard panes. Each of the three test types supported by the Eiffel Testing Framework requires certain unique information. Therefore, there is a unique sequence of wizard panes corresponding to each test type.
The '''New Eiffel test wizard''' creates new tests based on information entered by a developer on a series of wizard panes. Each of the three test types supported by AutoTest requires certain unique information. Therefore, there is a unique sequence of wizard panes corresponding to each test type.
The '''New Eiffel test wizard''' is invoked by clicking the '''Create new tests''' button ( [[Image:create new tests]] ) on the Eiffel Testing Framework interface toolbar.
The '''New Eiffel test wizard''' is invoked by clicking the '''Create new tests''' button ( [[Image:create new tests]] ) on the AutoTest interface toolbar.
@@ -17,7 +15,7 @@ The '''New Eiffel test wizard''' is invoked by clicking the '''Create new tests'
The first pane of the wizard allows you to choose which type of test(s) you wish to create.
[[Image: New test wizard screen 01 01]]
[[Image:AutoTest new test wizard 01 01]]
==The manual test wizard sequence==
@@ -32,7 +30,7 @@ Creating manual tests will involve using the following wizard panes. For a guide
In this pane you provide a name and cluster for the test class that will be created. You can also indicate that the created class should include redefined versions of features <eiffel>on_prepare</eiffel> (called during preparation for test execution) and/or <eiffel>on_clean</eiffel> (called during clean-up after test execution).
[[Image: New test wizard screen 02M 01]]
[[Image: AutoTest new test wizard 02M 01]]
===The "Test routine" pane===
@@ -40,7 +38,7 @@ In this pane you provide a name and cluster for the test class that will be crea
This pane gives you the ability to name the test, and to associate it with certain [[Create a manual test#About Tags|tags]].
[[Image: New test wizard screen 03M 01]]
[[Image: AutoTest new test wizard 03M 01]]
For tags, there are two entry boxes with '''Add''' buttons to their right. You can use the top entry box to create entries for a tag system that you have developed.
@@ -64,45 +62,43 @@ For extracted tests, the second pane to appear is the '''New test class''' pane.
===The "Extract application state" pane===
[[Image:New test wizard screen 03E 01]]
[[Image:AutoTest new test wizard 03E 01]]
This pane provides a depiction of the current call stack. Choose the routine or routines for which you want extracted tests to be created.
==The synthesized test wizard sequence==
==The generated test wizard sequence==
Creating synthesized tests will involve using the following wizard panes. For a guided tour of this process, see [[Using synthesized tests]].
Creating generated tests will involve using the following wizard panes. For a guided tour of this process, see [[Using generated tests]].
===The "New test class" pane===
[[Image:New test wizard screen 02S 01]]
[[Image:AutoTest new test wizard 02S 01]]
This pane appears slightly different for synthesized tests versus manual or extracted tests. The class name is actually a prefix upon which a numeric index will be appended. The index is depicted to the right of the entry box.
This pane appears slightly different for generated tests versus manual or extracted tests. The class name is actually a prefix upon which a numeric index will be appended. The index is depicted to the right of the entry box.
===The "Generate tests through AutoTest" pane===
[[Image:New test wizard screen 03S 01]]
[[Image:AutoTest new test wizard 03S 01]]
On this pane enter the information needed to produce synthesized tests. Use the '''Class or type name''' entry box in the lower left, along with the <code>+</code> and <code>-</code> buttons, to declare a list of one or more classes as target classes for synthesized tests.
On this pane enter the information needed to produce generated tests. Use the '''Class or type name''' entry box in the lower left, along with the <code>+</code> and <code>-</code> buttons, to declare a list of one or more classes as target classes for generated tests.
Other information necessary for creating synthesized tests is:
Other information necessary for creating generated tests is:
*'''Cutoff (minutes)''' -- How long the Eiffel Testing Framework will execute random invocations of the routines in the classes to be tested, specified in minutes.
*'''Cutoff (invocations)''' -- How long the Eiffel Testing Framework will execute random invocations of the routines in the classes to be tested, specified by invocation count.
*'''Routine timeout (seconds)''' -- How long the Framework will wait for completion of any invocation.
*'''Random number generation seed''' -- The seed used for random number generator used during the process. A value of zero instructs the Framework to use a seed derived from the system clock.
*'''Use slicing for minimization''' -- Designate '''slicing''' as the approach for minimizing the size of synthesized tests.
*'''Use ddmin for minimization''' -- Designate '''ddmin''' as the approach for minimizing the size of synthesized tests.
*'''Cutoff (minutes)''' -- How long AutoTest will execute random invocations of the routines in the classes to be tested, specified in minutes.
*'''Cutoff (invocations)''' -- How long AutoTest will execute random invocations of the routines in the classes to be tested, specified by invocation count.
*'''Routine timeout (seconds)''' -- How long AutoTest will wait for completion of any invocation.
*'''Random number generation seed''' -- The seed used for random number generator used during the process. A value of zero instructs AutoTest to use a seed derived from the system clock.
*'''Use slicing for minimization''' -- Designate '''slicing''' as the approach for minimizing the size of generated tests.
*'''Use ddmin for minimization''' -- Designate '''ddmin''' as the approach for minimizing the size of generated tests.
*'''Create html output''' -- Output history and statistics from the synthesizing process in html format.