Ticket #117 (closed task: fixed)

Opened 13 years ago

Last modified 13 years ago

Glue memory leak problems

Reported by: spascoe Owned by: selatham
Priority: desirable Milestone: ReFactored_Discovery_WebServices
Component: discovery Version:
Keywords: Cc:

Description

Glue is consuming a significant amount of RAM. In particular the tomcat process has grown on occasions to about 1/3 of total available memory (including swap). When swap comes into use, performance suffers.

We need to identify why tomcat consumes so much memory. This is suspected to be connected to eXist.

Also, glue is running 3 relational database systems (mysql, ingres and postgres) and apache. Do we need all these services?

Change History

comment:1 Changed 13 years ago by selatham

Regarding rdbs - we need mysql(security) and postgres(discovery). Can't think why we'd need ingres.

comment:2 Changed 13 years ago by selatham

  • Owner changed from lawrence to ko23

Bryan asked Kev O'Neill whether he knew more about problems with eXist which could be causing memory problems on glue.

comment:3 Changed 13 years ago by selatham

  • Component changed from NDG_Management to T01_Discovery

comment:4 Changed 13 years ago by ko23

  • Owner changed from ko23 to selatham
  • Type changed from issue to task

Update eXist to the 20060316 snapshot (go to  http://exist.sourceforge.net/). The benefits were pointed out in the last Metadata team meeting. Note that this works well under Tomcat, so a separate, non-Tomcat, installation is not necessary.

Not sure who looks after glue...

comment:5 Changed 13 years ago by selatham

  • Owner changed from selatham to pjkersha
  • Cc selatham removed
  • Component changed from T01_Discovery to NDG_Management

This may be sorted out by #69 "Upgrade NDG Discovery portal to latest eXist release".

comment:6 Changed 13 years ago by selatham

  • Owner changed from pjkersha to selatham
  • Component changed from NDG_Management to T01_Discovery
  • Milestone changed from ALPHA to ReFactored Discovery WebServices

comment:7 Changed 13 years ago by selatham

  • Status changed from new to assigned

Now should be sorted when do #481 "Upgrade exist on glue to latest version".

comment:8 Changed 13 years ago by selatham

Looks a lot better after upgrading exist. tomcat is running at around 12% memory. Before it was getting to about 35%. Monitoring.

comment:9 Changed 13 years ago by selatham

  • Status changed from assigned to closed
  • Resolution set to fixed

Monitored - seems OK. Closing.

comment:10 Changed 13 years ago by selatham

  • Status changed from closed to reopened
  • Resolution fixed deleted

Matt - I think we still need to look at this, since exist still has factory settings which even it suggests are not suitable for production environment. We haven't really tried hammering it with large queries yet either. I'll dig out the docs for next week & see what it recommends.

comment:11 Changed 13 years ago by mpritcha

Quoting recent post by Wolfgang Meier on Exist-open list:

If you can't use the client in embedded mode, you should at least make sure the server has enough memory. Check the -Xmx setting passed to the JVM in bin/startup.sh or bin/server.sh. By default, bin/startup.sh assigns no more than 128M to the server, which might not be enough for a mass-upload.

Another important parameter is the cache size used by the database itself. The cacheSize as configured in conf.xml (see <db-connection cacheSize=""/>) should not be more than half of the available memory set with -Xmx. Most likely this setting is causing the OutOfMemory? exceptions you see. For the last release, cacheSize was set to "96M". That's too much in comparison to the 128M given to the JVM in bin/startup.sh (we should fix that). So either increase -Xmx or decrease cacheSize to e.g. 48M.

comment:12 Changed 13 years ago by selatham

  • Status changed from reopened to closed
  • Resolution set to fixed

Increased -Xmx to 1G. Not had any memory leak problems recently. Closing.

Note: See TracTickets for help on using tickets.