I'm using Ubuntu 12.04 as a guest under Windows XP SP3 as a host (VirtualBox version: VirtualBox VM 4.2.2 r81494, Guest Additions: Version: 4.2.2 r81494 - if I'm looking into the right line in the log), for programming purposes. The program uses OpenGL and at one point it needs to create a fairly large mesh which is handled by glDrawElements method, with vertex buffer and index buffer passed to it (drawing mode is GL_TRIANGLE_STRIP). Everything works fine until the size of this mesh starts to exceed a certain limit. I wasn't able to determine it exactly (my code is a part of a bigger project and I'm using what was already done before me), but the vertex and index buffers take at least 60 MB of video card memory (could very well be some more). To be honest, I wasn't (and am not) sure that this issue has to do exactly with memory usage, but it is possible, given that absolutely the same code, but with smaller size of the mesh, works fine.
The issue is: Instead of drawing the mesh as it should, OpenGL draws a few chaotic rectangles filling up all the view (all while experiencing very low FPS which wasn't happening normally). Then the guest system stops responding, and within a couple seconds host system stops responding as well, freezing completely with the only choice left to reboot the machine.
To solve the problem I've maxed the video memory allocation to 128 MB (from previous variant 64 MB), and then, when it didn't help, to 256 MB (by manually editing the .vbox file), but it didn't help as well. Sometimes it's possible to notice for a fraction of a second that the mesh is drawn as it should be, but with colors of the vertices messed up (then the same scenario happens). Sometimes there's just enough time to be able to terminate the program before things start going completely south and save the machine from freeze.
I know I should also double-check my own OpenGL code to see if it has any problems, but it works absolutely fine with smaller meshes (the only thing changed in the code being the size parameter); plus it works fine on "physically installed" (not in VM) Ubuntu systems. Since increasing the allocated memory 4 times didn't help, I'm thinking maybe it's not directly related to the amount of memory available? I'm using 3D acceleration, and the physical memory on the graphic card is 512 MB. The card is quite an old ATI Radeon, HD 2600 XT, but it has latest drivers up to date from the official website.
Log file attached (VBox mesh crash.log). Don't mind stuff happening from 0:13:01 to 0:13:20, this was a testdrive of a smaller mesh (which didn't cause any crashes). The problematic moment (the described situation) starts at 0:14:40 in the log and ends at 0:15:02 (I managed to terminate the program in time).
I've also found VBox.log.2 in the logs folder which seems to be a log from the time when I didn't manage to stop the program in time so it freezed the PC completely (so the log, obviously, ends abruptly). Also find it attached (I had to rename it to VBox.log.2.log since .2 files aren't allowed).
Additionally, when I stopped the program and saved the system from freeze, I found the following output from OpenGL to the debug console of Qt Creator (in which the program was running):
Code: Select all
OpenGL Warning: vboxCall failed with VBox status code -54
OpenGL Warning: SHCRGL_GUEST_FN_WRITE_READ (37748776) failed with 0 fffffffe