[RDT-Dev] OOM Error with Experimental Indexer

[Werner Schuster (murphee)]

Running the current CVS Head, I'm getting OutOfMemory Errors, coming
from the Experimental Indexer Job;
I have some big-ish projects (rubinius, etc) in the workspace.

Has anyone seen this?

murphee

[Christopher Williams]

murphee,
I hadn't actually run into any OutOfMemory exceptions myself, but I had
been looking at/concerned about the indexer. I was seeing huge, huge amounts
of RAM consumed (I set eclipse to use up to 512M of RAM) using a profiler.
There were some bugs I thought I fixed fairly recently, but I was also
mistaken a bit by the behavior.
RDT has the same behavior as JDT: when you traverse the RubyModel (which is
what I do in the indexer), it stores RubyElements in the RubyModelCache. It
sets up the cache to overflow. It should check how much RAM is available at
startup, set hard limits on the cache and then grow until it hits those
limits. At that point it should start overflowing and cleaning up pointers
to free memory.

But, you might be facing a different memory situation which is that I
generate index keys for every unique IPath I hit. I generate a
SearchDocument and throw them in a HashMap keyed by the IPath. The
SearchDocument basically just holds a List of Strings each of whcih
represents an index key for a ruby element. Apparently there are enough
documents and keys generated to cause you troubles. To fix this, I'd need to
add more smarts to the indexer to write out indexes to files and clean up
the in-memory copies, etc.

How old is the snapshot you're using? Do the absolute latest builds fix it?
What is your RAM limit set to for Eclipse?

Thanks,
Chris

Post by Werner Schuster (murphee)
Running the current CVS Head, I'm getting OutOfMemory Errors, coming
from the Experimental Indexer Job;
I have some big-ish projects (rubinius, etc) in the workspace.
Has anyone seen this?
murphee
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Rubyeclipse-development mailing list
https://lists.sourceforge.net/lists/listinfo/rubyeclipse-development

[Werner Schuster (murphee)]

Post by Christopher Williams
But, you might be facing a different memory situation which is that I
generate index keys for every unique IPath I hit. I generate a
SearchDocument and throw them in a HashMap keyed by the IPath. The
SearchDocument basically just holds a List of Strings each of whcih
represents an index key for a ruby element. Apparently there are enough
documents and keys generated to cause you troubles. To fix this, I'd need to
add more smarts to the indexer to write out indexes to files and clean up
the in-memory copies, etc.
How old is the snapshot you're using?

I updated from CVS HEAD today;

Post by Christopher Williams
What is your RAM limit set to for Eclipse?

I ran it by launching it in the Eclipse debugger - I'm not entirely sure
what the default limit is.
I tried it again with Xmx set to 512 M and that seems to have solved it
- at least I don't get the problem again.
Two options:
- just mention this in a FAQ (Set Xmx to 512)
- or catch the OOM Error and just log it (I got an error that popped up);

[Christopher Williams]

murphee,

Post by Werner Schuster (murphee)

Post by Christopher Williams
What is your RAM limit set to for Eclipse?

I ran it by launching it in the Eclipse debugger - I'm not entirely sure
what the default limit is.
I tried it again with Xmx set to 512 M and that seems to have solved it
- at least I don't get the problem again.
- just mention this in a FAQ (Set Xmx to 512)
- or catch the OOM Error and just log it (I got an error that popped up);

I'd consider these bad fixes for a problem that can be fixed in code. We may
want to mention the bumping max memory somewhere, but most people will
likely never find where we put it (and besides increasing the max memory is
standard for working around OOM issues). You can change the eclipse.ini file
to make new defaults. (I think 3.1M6 introduced this along with higher
default memory limits).

BTW, 512M is just what I use because I happen to have like 30 eclipse
plugins in my RDT workspace. I would hope that most Ruby workspaces wouldn't
approach that level. I play around with RDT in a workspace with about 8
rails projects in it, and my profiler says that the experimental index holds
about 24 MB of index data in memory.

Just for my own curiosity: What is in your workspace? How may ruby/Java
projects? how many files?

Thanks,
Chris

[Markus Barchfeld]

Post by Christopher Williams
BTW, 512M is just what I use because I happen to have like 30 eclipse
plugins in my RDT workspace. I would hope that most Ruby workspaces
wouldn't approach that level. I play around with RDT in a workspace
with about 8 rails projects in it, and my profiler says that the
experimental index holds about 24 MB of index data in memory.

I got the OOM error, too. I had three rails applications with vendor
directories in my workspace. I ran a runtime-workspace and increased
memory to 512m which did not help, too (I added -Xmx512 to the VM
arguments, hope fully it really does what I expect). Only by closing one
of the rails projects I could finally get rid of the OOM error.
(Un-?)fortunately I can not reproduce the error with current HEAD
although I could not see any code change concerning the Indexer.

Thanks,
Markus

[Christopher Williams]

Markus,

Post by Markus Barchfeld

Post by Christopher Williams
BTW, 512M is just what I use because I happen to have like 30 eclipse
plugins in my RDT workspace. I would hope that most Ruby workspaces
wouldn't approach that level. I play around with RDT in a workspace
with about 8 rails projects in it, and my profiler says that the
experimental index holds about 24 MB of index data in memory.

I got the OOM error, too. I had three rails applications with vendor

Post by Markus Barchfeld
directories in my workspace. I ran a runtime-workspace and increased
memory to 512m which did not help, too (I added -Xmx512 to the VM
arguments, hope fully it really does what I expect). Only by closing one
of the rails projects I could finally get rid of the OOM error.
(Un-?)fortunately I can not reproduce the error with current HEAD
although I could not see any code change concerning the Indexer.

Well I'm hoping that this was an artifact of the problems in the
RubyModelCache that I mentioned earlier then since it cleared for you. I had
found memory problems/bugs there that I did fix. The Indexer just happened
to exercise the cache a lot so it unearthed the problem quickly.
The bug in question is that the RubyModelCache didn't have a specific cache
for ISourceFolderRoots (because until fairly recently we didn't even have
the notion of them). So they were getting thrown into a plain old HashMap,
and I believe holding onto all of their children. So we were effectively
storing all of the model in a HashMap that didn't overflow or clean up when
it approached the memory cap. There should have been a noticeable memory
difference between versions before my fix to the RubyModelCache file and
after.

Thanks,
Chris