This document will help you debug blocked portal activities, explaining how portal actitivies and CMFActivity are used in ERP5 as well as the steps you can undertake to trigger activities to finish properly.
Table of Contents
What are Portal Activities?¶
Portal activities are run by CMFActivity. Everything in ERP5 is a process and portal activities can be seen as the queue working through all processes triggered by user interactions, alarms and other sources.
You can access portal activities through the URL [your_instance_url]/portal_activities
Normally this listbox should be empty if your instance is "idle" and nobody is working on it. When your instance is being used, activities are added and removed once they are processed. However, when developing ERP5, adding and removing business templates and other activities may cause portal activities to block and stall.
If this happens, try any of the following steps to get your activities to clear. Note, deleting pending activities is NEVER a good idea, because activities represent processes which need to finish in order for user interactions to be completed. For example if you create a new person and save, the object is created and added to the index. If you would remove the process which adds the new object to the index, your object would be there, but you could not find it, because searching for it requires the object to be properly indexed. It will never show up anywhere.
Restarting Activities¶
Sometimes unblocking works by simply restarting activities. You can either trigger activities manually or via the SQL interface. Note that all blocked activities should be restarted once underlying errors have been found and removed.
Go to [your_instance_url]/manage_main (zope access needed) find erp5_sql_connection and select the tab test.
In the SQL command window, enter:
update message set processing_node = -1 where processing_node = -2;
update message_queue set processing_node = -1 where processing_node = -2;
Wait and see whether activities finish or block again due to the same or another error.
Note there are messages, which can be processed "standalone" and a message_queue, in which processes are depending on the outcome of previous processes. If one process throws an error, all dependend processes will also be blocked until the process throwing the error can finish.
Check for Errors¶
If your activities are stalled check for errors in the error log. Go to [your_instance_url]/error_log/manage_main (requires superuser/developer access rights). Oftentimes there is an error preventing an activity to finish. Check the stack trace and try fixing the error and restart a couple of activities manually to see whether they pass.
Resubscribe to Portal Activities¶
If you are running portal_component tests and your test throws an error, ERP5 unsubscribes portal activites. This means even fixing your error will not trigger any activities until you resubsribe.
To resubsribe switch to the zope interface (zope access needed). and select the load balancing tab.
Click the subscribe/unsubscribe button at the bottom to resubscribe and retrigger pending activities.
Set Correct Processing Node¶
Look what node is processing activities on the ERP5 interface. In the load balancing tab set this node as processing node in case it was reset to another node. (Note, most of the times only one or two nodes exist).
Check Webrunner Error Log¶
Another place to check for errors is directly inside the webrunner. Inside the SlapOS Webrunner Editor, open:
/instance/slappart[x]/var/log
Add the following:
tail -f zope-*.log
Add some space and restart the blocking activity in erp5. Watch what error pops up. Fix it and try to restart the activity again.
Restart Zope Service¶
When running portal component tests or cancelling them midway through it might be required to restart Zope. After restarting your erp5 instance will be gone for a few seconds, then it will be accessible again. See if that helped.
Restart in Theia¶
Find out the name of the service via:
slapos node status
Then restart it:
slapos node restart {the-name-of-your-service}
Restart in Webrunner¶
Go to the services tab, locate the Zope-0 service and click restart.
Clear caches¶
ERP5 caches a lot of data in different caches. When running portal component tests or cancelling them midway through, some cache files might not get properly updated, so on subsequent tests you keep using an outdated cache. Try clearing caches by going to [your_instance_url]/portal_caches/manage_main and clearing all caches. Note: Doing this on a production system may seriously slow down your application, so beware.
Missing Tables¶
When working with business templates a frequent problem blocking portal activities are missing tables resulting from business template definitions not being complete.
This can be fixed by going to the Zope interface [your_instance_url]/portal_catalog/manage_main selecting mysql and calling the method to create the missing table manually.
The missing table should be detectable in the error log. Should you receive a fatal reindex error on missing tables error, go to the properties tab while in portal_catalog and unset raise error on id change at the end of the page. Don't forget to reset it once you have created your table, reindexing is done and all activities have finished.
Recreate Tables¶
Still in the portal_catalog section, go to the advanced tab and click recreate tables. (don't klick the red button if it is available). WARNING: This is close to the nuclear option and should never be done on a live system (!).
Recreate Tables Manually¶
In case you do not have the methods to create tables in the Zope interface [your_instance_url]/portal_catalog/manage_main, you can try to create this tables manually. Go to [your_instance_url]/manage_main, click on erp5_sql_connection and open the Test tab. Here you are able to manually create the missing table. For example, this is how to create a email table :
CREATE TABLE `email` (
`uid` BIGINT UNSIGNED NOT NULL,
`url_string` varchar(255),
PRIMARY KEY `uid` (`uid`),
KEY `url_string` (`url_string`)
) ENGINE = InnoDB;
Don't forget to restart the activities when it's done.
ReindexAll¶
WARNING: Don't ever trigger this on a live system (!).
Delete all activities, then append /ERP5Site_reindexAll at the end of the url. Wait.
You could also clear the catalog by calling /ERP5Site_reindexAll?clear_catalog=1.
Note: When reindexing while logged in as ERP5 User and clearing the catalog you will effectively remove your logged in user from ERP5. Make sure you log out and back in as zope user before doing this to prevent loosing time wondering why you are logged in but cannot do anything.