Libresource server crash : unable to create new native thread

Comment of : Libresource server crash : unable to create new native thread

gdf — 2008-06-02T15:56:52+02:00

We confirm that this is also a problem with jonas 4_9_2. One of the advantages of the last version of jonas was that it solved this error message :

ProxyImpl.doReact : org.objectweb.joram.shared.excepts .StateException: Context 35 is closed or broken.

and certainly a couple of other bugs …

Comment of : Libresource server crash : unable to create new native thread

Sebastien Jourdain — 2008-06-02T13:10:37+02:00

So the problem come with JONAS 4.9.1 and Quartz.

Comment of : Libresource server crash : unable to create new native thread

gdf — 2008-06-02T11:42:36+02:00

OK, so here's what we have done :

create a script to simulate an activity on the forge
revert to jonas 4_7_6 on a test server
simulate activity on the test server during 10hours (Result : the increase in threads disappeared.)
revert to jonas 4_7_6 on our production server for 72hours (Result : the increase in threads disappeared or is much slower than initially.)

We advise not to use jonas 4_9_1 with libresource. In fact, we are currently testing jonas 4_9_2 on our test server, and the dreaded increase in threads is still present.

Comment of : Libresource server crash : unable to create new native thread

Sebastien Jourdain — 2008-05-29T12:17:15+02:00

Could you make a test with the Libresource installer on another database, to see if the behavior is the same with jonas 4.7.6 ?

Comment of : Libresource server crash : unable to create new native thread

gdf — 2008-05-29T11:21:00+02:00

While observing the server with "jconsole", we notice a HUGE number of threads named "Libresource Scheduler_Worker". All of them are grouped by packs of 10, and go along with a "Libresource Scheduler_QuartzSchedulerThread". The number of these threads is only increasing, and the console shows that most of them keep waiting indefinitely.

Our questions :

What's the use of the LibreSource Scheduler (knowing that there already exists a scheduler in Jonas) ?
Is there a timeout on these threads and how do we configure it ?

Comment of : Libresource server crash : unable to create new native thread

Sebastien Jourdain — 2008-05-28T18:22:10+02:00

To change the server :

download jonas 4.7.6
backup the database
download the EAR from our server
configure the jonas
start the server on the same DB.

Thats it.

Regarding the scheduler :

The scheduler used in libresource is like a cron
As I remember only the synchronization of remote site use this scheduler. But I thought you might have use it.
If quartz is switch off during a long time with a task that tries to start. then quartz keep starting this task in a new thread...

Comment of : Libresource server crash : unable to create new native thread

gdf — 2008-05-28T17:56:27+02:00

I do not understand what you mean by "look at the content from quartz". If you mean the "qrtz*" tables inside the libresource database, all of them are empty (except qrtz_locks that contains only 5 rows). If you mean doing something with the quartz scheduler, please could you explain a bit more as I am not an expert on this.
Changing the jonas server is not really easy for the moment, because we are still on an ongoing test that will last until the middle of June, and can not really be stopped. But it could be done if we can find some time to spend on it.

For reference, we are also monitoring the activity of a second forge that supports our developpements on the forge itself. The second forge exhibits also the same phenomenon, that is to say the steady increase of the number of "live threads". But as this forge is used by only a few people, the increase is rather slow.

Comment of : Libresource server crash : unable to create new native thread

Sebastien Jourdain — 2008-05-28T17:43:59+02:00

strange… We never had such problem. Did you try with jonas 4.7.6 ?

Comment of : Libresource server crash : unable to create new native thread

gdf — 2008-05-28T17:38:47+02:00

This server is under test here, but we are testing it with as an exercise with students, so the number of persons using the forge is rather large. This server has had multiple crashes since the beginning of our tests, but has never been stopped for more than a night.

It must be said that for our purposes, we have created a JMS java client that sends messages to the forge (to generate activities when subversion commits are made). This could be the culprit, but the number of messages sent is less than 200 over the 36hours of this crash.

Comment of : Libresource server crash : unable to create new native thread

Sebastien Jourdain — 2008-05-28T17:28:45+02:00

If so look at the content of the table from quartz. And drop the content or the table while the server is down. Then start the server again.