This visual guide has been created for learning and for teaching performance testing for

This visual guide is mostly useful to ERP5 developers who need to understand and performed performance testing,

to users who are willing to understand ERP5 and to marketing people who need to explain performance testing for ERP5.

Readers should first have a quick look at the illustration on the upper part of each page then read the short text bellow

the illustration carefully and associate each word written in bold to the corresponding item(s) in the illustration.

For example, the term Creative Commons License is written in bold because it defines the license of the above illustration.

Performance testing is about ensuring that the user gets the reply to his request in a timely fashion without considering concurrent accesses, and addressing bottlenecks.

Once performance testing has been done, scalability testing is then performed to address potential bottlenecks arising with concurrent accesses.

Scalability testing will be explained in a separate presentation.

When performing performance testing, first of all, a performance test plan has to be defined according to the most common tasks performed by users.

Secondly, tasks must be group together by relevancy, where each operation of a group defines precisely realistic and acceptable times elapsed for the round-trip from an user point of view.

Thirdly, a script must be written for each group gathering reproducible results.

These script usually rely on an existing tool or framework.

This presentation will explain how to perform these three steps by giving a simple example and will rely on erp5.utils.test_browser Python package as the performance testing framework.

In the following slides of this presentation, a simple example will be used where the user only add users.

This task is the most common tasks performed in this example, therefore this is where the performances matters the most.

As it is only composed of basic operation, they all should not be perceptible by the user.

In this example, there is only one group, namely adding persons, but generally there will be much more groups.

For example, if managing bugs is another common tasks performed by users, there could be another group about adding, modifying and deleting bugs.

Along with the common tasks being performed for this example, there are also the steps done by the user to achieve such tasks.

At this point, there is everything required to write the script to test the performance of these tasks.

The following slides will show how to write a script for this example after introducing briefly the performance testing framework written specifically for ERP5.

Before explaining how to write the scripts for the performance test plan described before, it is necessary to introduce the framework which is going to be used for performance testing.

mechanize is a Python package providing a stateful programming web browser.

Namely, it provides an API which allows to simulate what an user performs through a web browser by writing Python code. For example, it allows to open an URL, get the response, fill and submit forms, as well as supporting browser history and cookies. The Python package zope.testbrowser is actually an abstraction of mechanize with a special focus on testing by measuring the elapsed time and pystones for each request. Besides of that, it also provides a nice and easy to use object model for forms and links.

The Python package z3c.etestbrowser extends zope.testbrowser by allowing to parse an HTML page as an elements tree and use XPATH for example to easily selects elements within an HTML or XML document. It also provides HTML and XML normalisation to display such contents in a readable way.

erp5.utils.test_browser only provides helpers specific to ERP5, such as selecting an ERP5 module, a favourite, copying and pasting previously selected objects, or getting the number of remaining activities.

However, note that Javascript is not supported (because it would require a Javascript engine), which could be circumvented by calling the target URL of Javascript code for example. Anyhow, Javascript testing is more related to functional testing and could be performed with Windmill or Selenium for example

Before doing anything, you should have installed erp5.utils.test_browser package by running the following command line:

easy_install -f 'http://www.nexedi.org/static/packages/source/' erp5.utils.test_browser

This is just a basic script going to ERP5 homepage after instanciating Browser class from ERP5 benchmark module. The URL should always have a trailing slash. Note that you can give to open() method either a relative path to the ERP5 site (for instance, if you wish to open /erp5/person_module, you just have to pass person_module to open()) or an absolute URL (the former is generally the most commonly used).

In the next slides, the function benchmarkAddPerson() will be completed following the steps given in the slide before.

In ERP5, there is only one form, whose id is main_form. Therefore, we first get the main Form object

(as defined in zope.testbrowser package and extended by erp5.utils.test_browser package) and then select the module either by value or label

(in terms of HTML, it means selecting the by value or label within a and can be therefore used the same way for any ).

label is of course easier to use as it does not required looking at the HTML source code, however, it is preferable to use the value as it will always

the same whereas the label is more susceptible to change and is language-dependent.

Note that the given label value is searched as case-sensitive whole words within the option labels, and that the value is searched in the same way and

also if it is found at the end of the string excluding the query string.

Internally, it will actually select the field and then submit it, therefore the URL before executing this code will be

http://foo/erp5/ and http://foo/erp5/person_module/ after

As explained in the previous slide, you can either select an within a by value or label. Furthermore, you could also use the New button. Similarly, most of the buttons and menus can be activated through methods defined in the main form instance (from erp5.utils.test_browser package), such as Delete and Copy button by calling respectively submitDelete() and submitCopy() or Favourite by calling submitSelectFavourite() (which takes the same parameters as other submitSelect*() methods) Fill the form First, we check whether the person has been created by looking at the transition message (also known as the Portal status message) by calling getTransitionMessage(). Then we fill the fields for the first name and last name by getting the control by their name and setting the value which is going to be submitted. getControl() method could be used to get any field on the page either by name or label (the latter is usually easier to use, as shown before, but of course it does not really make sense to use it on a text field).

Similarly to the New button, we call submitSave() to save the modification made in the previous step and then check that the data has been actually saved properly.

submitSelectWorkflow() method is a bit different to other submitSelect*() methods for the value parameter because the workflow to be executed is given in the query string.

The script identifier (Base_viewWorkflowActionDialog in this example) can also be given through the parameter script_id which defaults to Base_viewWorkflowActionDialog. However, the label parameter is the same as the other methods.

Now confirm the dialog form by calling submitDialogConfirm(), there is also a submitDialogCancel(), and finally check that the validation is successful.

After this point, it could also be relevant to check for the number of activities (by calling getRemainingActivityCounter() method) to ensure that it decreases after each call of the script main function.

This is the first part of the final script which shows how to measure the time elapsed upon creation, save and confirmation of the person.

Basically, any method can be prefixed by time to get the number of seconds the request took.

As explained in a previous slide, it is essential to perform several iterations to get pertinent results.

In this example, only 20 iterations will be done, but in real-world performance tests, there should be much more of course

The second part of the final script calls the main function 20 times and gathers the results.

For each operation (create, save and validate), the average time is calculated and finally displayed.

The final results shows that the average of each operation is approximatively half the acceptable timeframes defined at the beginning.

Shortcomings:

• If there are much more iterations, perhaps you should add a sleep() after each call as there might be many activities left which could slow down the next iteration.
• The average value may not be enough to measure the performances, therefore other statistic values could be calculated (such as standard deviation).
• Measuring the performance in seconds is fine when running the performance script on the same machine but for scalability testing, it will not be enough as it is dependant on the processor being run, so using pystones will be more appropriate.

z3c.etestbrowser documentation

zope.testbrowser documentation

You can download a tarball of erp5.utils.test_browser package on Nexedi website: nexedi.org/static/packages/source. You can then generate the documentation as explained in README.txt. There is also an example in examples/ directory.

z3c.testbrowser and especially zope.testbrowser are extensively documented with examples on their respective project pages, and, as erp5.utils.test_browser follows exactly the same API, it is worth having a look