Page 1 of 1

VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client

Posted: 2. Aug 2014, 01:13
by llagos
Hello Users,

I'm getting some weird behavior. I have several RHEL 6.5 64bits guests. Host is Win7 64bits.

They were all working fine, after I needed to increase de memory for one of them. I raised it from 2G to 3G. I started the guest, the login windows appears (gnome), I log in with some user, open a terminal, and then, I type any command, and hit ENTER.

Right after I hit ENTER, the sessions ends abruptly, and I am back again to the login GUI. When I logged back in again, the redhat reporting system showed two errors saying

xorg-x11-server crashed.

I tried again this time increasing the RAM for 3G to 4G this time, and again the same behavior. I tried logging in a few times, right now it's not resetting, so I could send a "dmesg" command, where I can see these messages:

VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18
VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Client=18

After the sucessful dmesg, I just cd to another dir, but when I hit ENTER, again the same problem.

I don't know if the ram change may have affected this... but this started after that.

Any idea what could be wrong?

Regards,

Re: VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Clie

Posted: 2. Aug 2014, 01:22
by socratis
llagos wrote:I raised it from 2G to 3G.
llagos wrote:I tried again this time increasing the RAM for 3G to 4G
Did you ever try to lower it to 2GB to see if it changes the situation? Just to make sure that the memory increase is not the culprit?

Re: VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Clie

Posted: 2. Aug 2014, 01:59
by llagos
Hi,

Worst. Now, right after I hit ENTER when entering the password on the gnome login gui, it turned black, resize a bit smaller, and again the login gui.

I noticed, when it goes black to a back screen, terminal mode, the bottom line has some strange characters, as when you type special chars (non-printable).
strange chars at the bottom
strange chars at the bottom
linux guest resetting.png (56.46 KiB) Viewed 7208 times
Don't know what can it be, but it's definitively something graphical. I ssh from another guest, and it works ok. The other guest is the same config, from same iso, same rpm versions, same everything.

Thanks a lot,

Re: VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Clie

Posted: 2. Aug 2014, 13:35
by mpack
Provide a VM log file for one of the problem sessions. Minimum information needed for assistance.

Re: VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Clie

Posted: 4. Aug 2014, 18:41
by llagos
Hi,

Here are the logs from the hosts.

thanks,

Re: VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Clie

Posted: 4. Aug 2014, 18:51
by llagos
Checking the aprox time I hit ENTER, I can see this line which looks suspicious.

UIFrameBufferQImage::resizeEvent: Resizing to FALLBACK buffer due to format is invalid..

Really don't know but it's the only thing I see that mentions some kind of error.

Re: VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Clie

Posted: 6. Aug 2014, 17:31
by llagos
mpack wrote:Provide a VM log file for one of the problem sessions. Minimum information needed for assistance.
Hi, Have you been able to find something to this issue?

thanks!

Re: VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Clie

Posted: 6. Aug 2014, 18:48
by mpack
Sorry, somehow I missed your previous response.
VBox.log wrote: 00:00:05.413361 VRamSize <integer> = 0x0000000000c00000 (12 582 912, 12 MB)
I don't know if this is the problem, but 12MB seems a tad low for a 1280x804x32bpp display. That's only three frames worth. Is there no warning in the settings dialogs about this?

p.s. It isn't necessary to provide three logs when only one is asked for. It just creates extra work.

Re: VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Clie

Posted: 9. Aug 2014, 00:01
by llagos
Those 12MB were the default. And according to recommendation on the manual:

A rough estimate is (color depth / 8) x vertical pixels x horizontal pixels x number of screens = number of bytes.

So, 32/8*1280*800 = 4M...

How much should it be then?

p.s.: I got it, thanks-.

Re: VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Clie

Posted: 9. Aug 2014, 12:00
by mpack
Yes, 12MB was the default, so what? The default display size was considerably smaller too, but you changed that.

I asked if it was giving you a warning (look for message in red on the settings dialogs), which I'm pretty sure it will. Tune the VRAM setting until this warning goes away - or else just bang it up to 64MB say. Your host has plenty of free RAM.

Re: VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Clie

Posted: 18. Aug 2014, 23:54
by llagos
Thanks mpack. I resumed working on this today. I'm sorry my maths is obviously wrong, just don't know how to calculate it correctly.

Anyway, I changed video RAM to 64M, started the VM guest, and .... got a blue screen... with kernel dump and everything! (win7pro 64bits).

This, I think it's quite serious. No matter the problem, any program should give warnings, or errors, but not a crash like this.

There is no log for today, so I assume the crash was well at the beginning.

I'll try again now, but just wanted to write this, in case it crashes again.

Regards,

Re: VBoxGuestCommonIOCtl: HGCM_CALL: Invalid handle. u32Clie

Posted: 18. Aug 2014, 23:56
by llagos
BTW, I don't know how did I changed the resolution. I don't see that setting in the VM settings.