Import of Business Template in unit test¶

Let's take the erp5_base bt with testERP5Base, it takes :

    - by default, 7.9s for importing, 14.2s fully installed, 65s for 
      testERP5Base
    - by activating zexp files (replace the "if 0" with "if 1" in BT.py)
      3.8s for importing, 9.9s fully installed, 54s for testERP5Base (all zexp
      available)
    - by not importing skins files (only import required ones at run time)
      2.3s for importing, 7.1s fully installed, 41s for testERP5Base
    - by not importing skins file and using zexp files
      1.3s for importing, 6.1s fully installed, 38s for testERP5Base
      (all zexp available)
    - by not using the filesystem anylonger and importing
      directly bt5 files as ZEXP from an existing instance, 0.3s for importing
      5.1s fully installed.

Missing tests¶

    - by default import, then only installing needed skin files (from bt5)

JPS: lazy install of skins seems to provide a speed gain of 1.3s. However, I do not understand why importing time for "not importing skins file and using zexp files" is lower than importing time for "by activating zexp files"

Seb: This is because the code (wich is not well designed) that create zexp files cut the bt5 into many zexp files (for every python script, every page template...). So the test was done by importing only one part of the bt.

Make activities faster¶

It could be possible to make activities runing on several cores.

--save speed optimisation ideas¶

- import bt5 in one big zexp for all bt5 that didn't change

- optimise XML parsing by any way (cache/MD5)

- do lazy import, so import on bt5 only needed stuff

- do lazy installation, so do not install in each bt5 what we don't need

- speed up indexing, in general, and may be with some trick (like only for some unit test, do not catalog portal types,...). One possible trick could consist in dumping the catalog of an existing installed instance then delete any UIDs which do not relate to any existing path. However, this is not really a safe approach.

- it must also be considered to make lazy installation persistent, in order to not reimport while using load for many unit test -> use ZEXP import approach from another instance, it is both faster and fully consistent with ERP5 ZODB based configuration approach

- HOW TO MAKE IT COMPATIBLE WITH --update-only ?

Future directions¶

The fastest test execution can be obtained by

1. Importing already parsed and built ZEXP from a running ERP5 instance instead of parsing text files on the filesystem (this also has the advantage of eliminating the issue of synchronization between files and ZODB).

2. Adding a certain amount of 'lazy install' so that certain parts of an already imported business template which contains numerous objects are only installed if needed. Skins are a good candidate.

3. Reduce the number of objects which are indexed, whenever this is not needed for a test (ex. do not index portal type actions, do not index python scripts).

4- Accelerate catalog, by any possible means, including by running tests on a ZEO cluster with concurrent activities.

Work Progress¶

Reduce the number of objects which are indexed : this is already done. By default portal types are not indexed. It can be indexed again with option --enable_full_indexing=portal_types for runUnitTest

Optimise XML parsing by any way. This is already done by storing xml file in zexp format in a gdbm storage. You can use ERP5_BT5_CACHE=1 environment variable when runing test.

Hyper Open Edge Cloud

Unit Test Speed