Resource Menu

Id : 134
Type : BUG
Reporter : gdf
Assignee :
Sebastien Jourdain
Priority : MAJOR
Creation Date : 28 May 08 05:09
Last Update Date : 02 Jun 08 03:56
Resolution : Won't fix
Libresource server crash : unable to create new native thread
The Libresource server (along with its JONAS counterpart) are subject to repetitive crashes on our server.
  • Version of Libresource : 2.5-rc2 (not upgraded yet)
  • Version of JONAL : 4.9.1
  • Java version : 1.6.0_06
  • System : Solaris
The conditions in which we are undergoing these crashed are the following :
  • server running for something like 36 hours
  • 35 different projects on the server (but only 10 are active)
  • about 100 users
  • all projects are using a subversion resource, and most of the activity consists of subversion commits.
The system crashes and the JONAS log file shows multiple SEVERE-rated errors, most of which are entitled :

java.lang.OutOfMemoryError: unable to create new native thread

jonas.log 378 ko. May 28, 2008 5:14 PM Go to file
memory_1.jpg 95 ko. May 28, 2008 5:15 PM Go to file
memory_2.jpg 101 ko. May 28, 2008 5:15 PM Go to file

Upload a new file :   
14 comment(s)
comment by gdf - 02 Jun 08 03:56 PM We confirm that this is also a problem with jonas 4_9_2. One of the advantages of the last version of jonas was that it solved this error message :

ProxyImpl.doReact : org.objectweb.joram.shared.excepts .StateException: Context 35 is closed or broken.

and certainly a couple of other bugs …

comment by Sebastien Jourdain - 02 Jun 08 01:10 PM So the problem come with JONAS 4.9.1 and Quartz.
comment by gdf - 02 Jun 08 11:42 AM OK, so here's what we have done :
  • create a script to simulate an activity on the forge
  • revert to jonas 4_7_6 on a test server
  • simulate activity on the test server during 10hours (Result : the increase in threads disappeared.)
  • revert to jonas 4_7_6 on our production server for 72hours (Result : the increase in threads disappeared or is much slower than initially.)
We advise not to use jonas 4_9_1 with libresource. In fact, we are currently testing jonas 4_9_2 on our test server, and the dreaded increase in threads is still present.
comment by Sebastien Jourdain - 29 May 08 12:17 PM Could you make a test with the Libresource installer on another database, to see if the behavior is the same with jonas 4.7.6 ?
comment by gdf - 29 May 08 11:21 AM While observing the server with "jconsole", we notice a HUGE number of threads named "Libresource Scheduler_Worker". All of them are grouped by packs of 10, and go along with a "Libresource Scheduler_QuartzSchedulerThread". The number of these threads is only increasing, and the console shows that most of them keep waiting indefinitely.

Our questions :

  • What's the use of the LibreSource Scheduler (knowing that there already exists a scheduler in Jonas) ?
  • Is there a timeout on these threads and how do we configure it ?
comment by Sebastien Jourdain - 28 May 08 06:22 PM To change the server :
  • download jonas 4.7.6
  • backup the database
  • download the EAR from our server
  • configure the jonas
  • start the server on the same DB.
Thats it.

Regarding the scheduler :

  • The scheduler used in libresource is like a cron
  • As I remember only the synchronization of remote site use this scheduler. But I thought you might have use it.
  • If quartz is switch off during a long time with a task that tries to start. then quartz keep starting this task in a new thread...
comment by gdf - 28 May 08 05:56 PM
  1. I do not understand what you mean by "look at the content from quartz". If you mean the "qrtz*" tables inside the libresource database, all of them are empty (except qrtz_locks that contains only 5 rows). If you mean doing something with the quartz scheduler, please could you explain a bit more as I am not an expert on this.
  2. Changing the jonas server is not really easy for the moment, because we are still on an ongoing test that will last until the middle of June, and can not really be stopped. But it could be done if we can find some time to spend on it.
For reference, we are also monitoring the activity of a second forge that supports our developpements on the forge itself. The second forge exhibits also the same phenomenon, that is to say the steady increase of the number of "live threads". But as this forge is used by only a few people, the increase is rather slow.
comment by Sebastien Jourdain - 28 May 08 05:43 PM strange… We never had such problem. Did you try with jonas 4.7.6 ?
comment by gdf - 28 May 08 05:38 PM This server is under test here, but we are testing it with as an exercise with students, so the number of persons using the forge is rather large. This server has had multiple crashes since the beginning of our tests, but has never been stopped for more than a night.

It must be said that for our purposes, we have created a JMS java client that sends messages to the forge (to generate activities when subversion commits are made). This could be the culprit, but the number of messages sent is less than 200 over the 36hours of this crash.

comment by Sebastien Jourdain - 28 May 08 05:28 PM If so look at the content of the table from quartz. And drop the content or the table while the server is down. Then start the server again.
comment by Sebastien Jourdain - 28 May 08 05:26 PM I've looked at your trace and it seems to occur when the scheduler try to execute a job. Did you stop the server for a long time, some times ago ?

comment by gdf - 28 May 08 05:23 PM We have not installed from the installer, since it was not working for us. We took the LS-2.5-RC2 files, and installed from sources. Everything (for libresource) has been compiled locally on the machine. We first thought that this was a problem of memory, and we increased the options to the following :

JAVA_OPTS=" -server -Xmx768m -Xms768m -Dfile.encoding=UTF-8"

comment by gdf - 28 May 08 05:20 PM Added some files for a better understanding :
  • jonas.log (log of the jonas server)
  • memory_1.jpg and memory_2.jpg (screenshots of the jconsole application for the jonas server, during the crash documented in jonas.log and another crash)
During both crashed, the number of threads increases steadily until it reaches a value of approximatively 3100. Then, it becomes impossible to create new threads. If nobody interferes, the server crashed by itself after a long series of SEVERE errors. In the second case, the decision has been taken to restart the server, but the situation was similar.

If we look precisely inside the JConsole, then there is a huge number of threads called "Libresource Worker Thread" inside the thread pool. We were hoping that these threads would terminate when the forge was not busy (during the night), but this is not the case.

comment by Sebastien Jourdain - 28 May 08 05:16 PM How did you installed Libresource ?
  • from installer => The ear hasn't been generated with the same jonas version as yours
  • from EAR => The ear hasn't been generated with the same jonas version as yours
  • from sources
Did you start jonas with more than 128 Mb ?

Do you have a bigger trace ?

Post a comment :


Wiki Syntax

Simple text filters

MnemonicRenderingComment
1 Title a style 1 heading (major heading)
1.1 Title a style 1.1 heading (minor heading)
- text a list item (with -), several lines create a single list
* text a list item (with *)
1. text an enumerated list
__bold__boldsimple bold text
~~italics~~italicssimple italic text
--strike--strikestrike through text
(empty line) produces a new paragraph
\\
 creates a line break, please use sparingly!
[/documentation] or [../documentation] or [/documentation|Doc]Like a standard linkinternal link to another document
http://www.libresource.org/http://www.libresource.org/creates a link to an external resource, special characters that come after the URL and are not part of it must be separated with a space.
\XXescape special character X (i.e. {)
---- creates a horizontal rule

Macros List

Macro syntaxe: {macroName:key1=param1|key2=param2|...}

MacroDescriptionParameters
apiGenerates links to Java or Ruby API documentation.1: class name, e.g. java.lang.Object or java.lang.Object@Java131
2: mode, e.g. Java12, Ruby, defaults to Java (optional)
asinGenerates links to DVD/CD dealers or comparison services. Configuration is read from conf/asinservices.txt1: asin number
blinkBlink a text.the text
bookmarksDisplay the user bookmarks.none
boxCreate a box in order to change the basic disposition.1. the box title (optional)
2. the floating behaviour (left or right) (optional)
3. the height of the box (px,%) (optional)
4. the width of the box (px,%) (optional)
childrenDisplay the current children list.1. uri= ...(optional) where you list the resources, the default uri is the current
2. groupby=… (optional) to group resources.The following value are available : type or owner or uri or creation or update or name
3. sort=… (optional) to sort resources. The values available are : type or owner or uri or creation or update or name
4. creation=...(optional) allows you to filter on the creation date. The values available are: Date (dd/MM/yyyy) or '+'/'-'/'!' followed by a Date(dd/MM/yyyy): allows to keep superior/inferior/exclude date
5. update=...(optional) allows you to filter on the modification date. The values available are: Date (dd/MM/yyyy) or '+'/'-'/'!' followed by a Date(dd/MM/yyyy): allows to keep superior/inferior/exclude date
6. type=...(optional) :allows you to filter resources with its type. The values available are : Project, Template, Timeline, Page, Repository, File, Forum, Thread, Message, Group, Synchronizer, BugTracker, Issue, Site, Survey, Form, DropBox
7. owner=...(optional):allows you to show only resources belonging to one user or many users.
8. limit=...(optional): allows you to limit the displayed resources
9. depth=...(optional): allows you to fix the depth of listed ressources
10. view=...(optional): to fix the view of displayed resources. The view available are : flat (default view) or count(to display the number of listed ressources) or tree(display resource following tree viewer)
codeDisplays a chunk of code with syntax highlighting, for example Java, XML and SQL. The none type will do nothing and is useful for unknown code types.1: syntax highlighter to use, defaults to java (optional)
helloSay hello example macro.1: name to print
htmlBy pass wiki engine to allow HTML code in contentthe html block
api-docsDisplays a list of known online API documentations and mappings.none
file-pathDisplays a file system path. The file path should use slashes. Defaults to windows.1: file path
imageDisplay an image.1. the image URI
2. align (right ,left or center) (optional)
3. width (optional)
4. height (optional)
inter-wikiDisplays a list of known InterWiki mappings.none
isbnGenerates links to book dealers or comparison services. Configuration is read from conf/bookservices.txt.1: isbn number
linkGenerate a weblink.unexplained, lazy programmer, probably
list-of-macrosDisplays a list of available macros.unexplained, lazy programmer, probably
loginBoxDisplay a login box.none
mailtoDisplays an email address.1: mail address
menuInclude the menu of the specified URI1. The uri for the menu to be include
newsDisplay a message thread as news.1. the thread URI
2. Max (optional)
preAllow rigid text style. (Keep text formating)none
printFieldDisplay a node forms fields values.1. (uri=) The uri on which you want to get the form value (Optional, by default=.)
2. (print=) Print or not the field label and the field value (Optional, by default=label, value, edit)
3. (field=) Print or not the specified field (Optional, by default will print all)
quoteDisplay quotations.1: source (optional)
2: displayed description, default is Source (optional)
quoteMessageQuote a message in another message by referencing only the message URI.1. The message URI to quote.
resourceInclude the view of a LibreSource resource.1. the uri of the resource (mandatory)
2. the view (optional)
3. the action (optional)
4. noaction (optional)
rfcGenerates links to RFCs.unexplained, lazy programmer, probably
rssInclude an RSS feed.1. the url of the rss feed (mandatory)
searchBoxDisplay a search box.1. the URI pattern
subProjectsDisplay the current subProjects list.none
summaryCreate the summary of a wiki page.1. the page URI (optional)
surveyDisplay results of a survey.the survey URI
tableDisplays a table.none
tableFormDisplay a table listing form results.1. (uri=) The uri on which you want to get the form value (Optional, by default use the local path)
2. (print=) Print additinal features such as a link to the data (goLink) or a link to get a CSV file (csvExport) (Optional, by default print all)
3. (field=) The list of field to show (Optional by default all)
timelineDisplay the first items of a timeline.1. the timeline URI
2. Max items (optional)
userGenerate a link to the current user page, according 'action'.1. the action (optional)
2. the link label (optional)
workspacesDisplay the user workspaces.none
xrefGenerates links to Java Xref source code.1: class name, e.g. java.lang.Object or java.lang.Object@Nanning
2: line number (optional)

Smiley

:-):-(;-):-P:-D:-[:-/:-*>:o8-):-$:-!:'(:-X =-O