CentOS 7 freezing since 8 months ago and I haven't found a solution yet

Discussions about using Linux guests in VirtualBox.
Post Reply
Rodrigo Gomes
Posts: 29
Joined: 19. Jul 2016, 13:19

CentOS 7 freezing since 8 months ago and I haven't found a solution yet

Post by Rodrigo Gomes »

Hello guys,

I have already reported these freezes on my guest machine running CentOS 7 at other times in the forum. But until today I did not get a solution, absolutely nothing solve my problem.

I already decreased CPU usage, I already decreased memory usage, not resolved.
The host machine is not used at all, just to keep the guest machine turned on. The memory and CPU usage of the host machine is minimal, the only standard for maintaining Windows.

The guest machine also runs very lightly, most often using only 1/2 of the available cores.
RAM memory is idle on both the guest machine and the host machine, on average 3GB on the host and 6GB on the guest.

These freezes do not happen when the use of hardware resources is high. In fact, after 8 months studying these freezes, there is no standard for this.

But I have to say, before I put more cores and memory in the guest machine and the freezes were much more frequent.
But today, after lowering everything, I do not understand why it still freezes.
My host machine is literally half the idle hardware.

I am using an outdated version of VirtualBox at this time because I park in the least freeze version. But all versions freeze, some more and some less, is extremely inconsistent. I don't even know if the version changes anything.

Well, I'm completely lost on this, I've done everything I could.
It was 8 months with this guest machine freezing from time to time.

Things I've Done:
- Decrease the use of cores by the guest machine
- Decrease the use of RAM
- Monitor whether high usage of any hardware element causes this freeze (disk, network, RAM, CPU, etc.)
- Disabling the GUI on the guest machine
- Disabling the audio driver in the guest machine through VirtualBox
- Disabling the USB Driver in guest machine by VirtualBox
- Test different configurations in guest machine through VirtualBox
- Install and remove the extensions package

Sorry for the long text, I wanted to explain all the details.
And sorry if something seems confusing, I speak another language that is difficult to transcribe into English.

Host:
Windows Server 2012 R2 Standard - 64 bits
32GB RAM
Intel Xeon E5-1620 v3
VirtualBox version 5.1.8 r111374 with Extension Pack

Guest:
CentOS Linux release 7.3.1611 (Core) - 64 bits
Guest additions installed
3D or 2D is not enabled
Attachments
logs.zip
(70.64 KiB) Downloaded 27 times
Rodrigo Gomes
Posts: 29
Joined: 19. Jul 2016, 13:19

Re: CentOS 7 freezing since 8 months ago and I haven't found a solution yet

Post by Rodrigo Gomes »

Note:
I use VirtualBox to virtualize a server used for shared hosting. Because it is much easier to maintain, migrate between dedicated servers and backup the entire system.

And due to the freezes, I already had the database corrupted once.

Should not VirtualBox be used for this purpose?
Should I have resorted to other softwares like VMware?
mpack
Site Moderator
Posts: 39156
Joined: 4. Sep 2008, 17:09
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Mostly XP

Re: CentOS 7 freezing since 8 months ago and I haven't found a solution yet

Post by mpack »

Your host has 4 cores, and you've allocated all 4 to the VM. This can cause problems. The RAM allocation looks high as well.
Rodrigo Gomes
Posts: 29
Joined: 19. Jul 2016, 13:19

Re: CentOS 7 freezing since 8 months ago and I haven't found a solution yet

Post by Rodrigo Gomes »

mpack wrote:Your host has 4 cores, and you've allocated all 4 to the VM. This can cause problems. The RAM allocation looks high as well.
Decreasing this number do not solve the problem as well.

In addition to the RAM that Windows already uses, there is still 3GB (almost 4) idle. This is is not enough? :?

As I said before, the host machine exists only to run this guest machine, all its hardware is used only for guest machine and Windows itself.
And freezing does not happen when resource use is high. Freezing happens most of the time when memory and CPU usage is minimal!
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: CentOS 7 freezing since 8 months ago and I haven't found a solution yet

Post by socratis »

I got to say right from the beginning that this is a really hard problem to replicate, so we'll have to go purely on intuition, unless there's a lot of debugging information recorded that might reveal something.

Having said that, everything seems "normal" up to the 844 hours that this "monster" of a VM is running. And then suddenly a freeze. Nothing before that point indicates a problem or something out of the ordinary. And 35 days of up-time is a pretty good record for a Windows computer. Does the Windows log show anything at the time that the VM freezes?

Finally one other idea is to disable the KVM paravirtualization option, and either go with "Legacy" or with "None". See if that helps. I have no clue if it will or not, but it's a relatively new addition to the VirtualBox arsenal and there might be some border cases where it might misbehave for unknown reasons to us common mortals.
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Rodrigo Gomes
Posts: 29
Joined: 19. Jul 2016, 13:19

PAE/NX for CentOS 64 bit

Post by Rodrigo Gomes »

I'm disabling some features that might be unnecessary and / or cause problems on my guest machine.

The PAE/NX is enabled by default, but I wonder if it is needed on a CentOS 7 64-bit?
Is it recommended to keep it enabled or disabled in this case?
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: CentOS 7 freezing since 8 months ago and I haven't found a solution yet

Post by socratis »

Since this is a hard one to replicate, it's a long one to replicate, it's taken you several months of battles to see what's going on, here's what I would suggest: enable debugging at some level.

I'll try to ask the developers if they have anything to suggest, i.e. what areas of the debugging they'd like to see enabled for this attempt and if they have any suggestions. You might even have to open a ticket report.

Did you see anything out of the ordinary in your Windows host's Event Viewer BTW?
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Rodrigo Gomes
Posts: 29
Joined: 19. Jul 2016, 13:19

Re: CentOS 7 freezing since 8 months ago and I haven't found a solution yet

Post by Rodrigo Gomes »

Hello socratis,

I checked the Windows logs and it has absolutely nothing strange. At the time the guest machine froze did not generate any log.

I went beyond and also checked the CentOS logs, also did not generate anything strange before freezing.
I have munin installed on the guest machine, everything was up to standard and the machine had many of idle resources at the time.

I've already enabled CentOS debugging mode before, but I also could not get more information from what happens before freezing.

But I would like to inform you that I have already tested this guest machine on a host machine with CentOS. In an attempt to see the problem solved by running CentOS as host and guest. But that did not solve either.
I just have not tested on other hardware because I rent a dedicated server and that would be something very difficult to do.
Having said that, everything seems "normal" up to the 844 hours that this "monster" of a VM is running. And then suddenly a freeze. Nothing before that point indicates a problem or something out of the ordinary. And 35 days of up-time is a pretty good record for a Windows computer. Does the Windows log show anything at the time that the VM freezes?
Do you think the virtualbox is not ideal for use with servers?
Finally one other idea is to disable the KVM paravirtualization option, and either go with "Legacy" or with "None". See if that helps. I have no clue if it will or not, but it's a relatively new addition to the VirtualBox arsenal and there might be some border cases where it might misbehave for unknown reasons to us common mortals.
I also tried disabling the "KVM paravirtualization" and the problem seems to have gotten worse.
And disabling the PAE/NX, do you think that will make any difference?

I want to delete this guest machine and install it all again, but once you prepare a server, it is very difficult to start a new one, especially when it is already in use.

I thank you so much for seeing this for me. I've had a lot of headaches over this and I've exhausted all my knowledge to find a solution.
And I can not give more information, because I do not see anything in the logs, the machine just freezes. And I also understand how difficult it is to reproduce this problem.

Whatever you ask of me I'll be willing to do to find out more information about it.
But maybe we have to wait a month to see it happen again.
Rodrigo Gomes
Posts: 29
Joined: 19. Jul 2016, 13:19

Re: CentOS 7 freezing since 8 months ago and I haven't found a solution yet

Post by Rodrigo Gomes »

After the guest machine started to freeze a lot, I started annotating each freeze, below are my notes if this is useful in any way:

VirtualBox v=5.0.24r108355 stable (test since 7/20/16)
- Unstable on 08/24/16
- Unstable on 07/10/16 - Guest machine freezes but has no errors or warnings

- Test started on 07/10/16: guest machine running without being in background, guest configured with less memory and CPU
- Test Feedback 11/11/16: guest machine so far has not frozen

--- New test started 12/11/16: VirtualBox updated to 5.1.8r111374
--- Test Feedback 10/12/16: guest machine so far has not frozen
--- Bug probably resolved with: guest machine running without being in background, guest configured with less memory and CPU
--- Unstable on 12/16/16 - the guest machine only froze, probably because of the intensive use of multiple VMs, confirming the previous theory regarding memory and CPU usage (I was running another VM to do some testing this day)
--- Unstable on 1/16/17 - guest machine just froze, no apparent reason, GUI was started
--- Unstable on 01/18/17 - guest machine just froze, no apparent reason, GUI was NOT started, 4am in the morning with idle processor and idle memory - windows checked for updates at the SAME moment the guest machine froze, possible Lack of memory

---- Test started on 01/20/17: guest configured with less memory and CPU
---- Unstable on 02/02/17 - guest machine only froze, no apparent reason, GUI was started, 6pm with idle processor and idle memory - windows NO checked for updates at the same time the guest machine froze
---- Test Feedback 3/16/17: guest machine so far has not frozen, a possible solution is the addition of a running task every minute on windows
---- Unstable on 3/16/17 - guest machine only froze, no apparent reason, GUI was NOT started, 2 PM, with idle processor and idle memory
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: CentOS 7 freezing since 8 months ago and I haven't found a solution yet

Post by socratis »

Rodrigo Gomes wrote:I've already enabled CentOS debugging mode before, but I also could not get more information from what happens before freezing.
I was thinking more in terms of enabling VirtualBox debugging, but it wouldn't hurt doing it at the CentOS level as well...
Rodrigo Gomes wrote:I have already tested this guest machine on a host machine with CentOS. In an attempt to see the problem solved by running CentOS as host and guest. But that did not solve either.
Excellent thinking! Take one parameter out of the equation, the host OS. Can I assume that you tried to run that CentOS physical system with the configuration that you have for the VM, i.e. replicate the CentOS VM functionality in the physical CentOS system? That would also take out the "workhorse" OS that's actually doing the job (CentOS), leaving for sure only one suspect; VirtualBox.
Rodrigo Gomes wrote:Do you think the VirtualBox is not ideal for use with servers?
I wouldn't personally have a clue. But if it's any help, and if I'm not mistaken, these forums, the ticket database and everything VirtualBox that you see publicly runs in a VM.
Rodrigo Gomes wrote:And disabling the PAE/NX, do you think that will make any difference?
I really couldn't tell. Theoretically the PAE shouldn't even be needed in a 64-bit system, but the developers have them enabled for several of them: Other-64, Win-64 (that's the generic template only, not specific versions), RedHat-64, Oracle-64, OSX*. So, from the major Linux/BSD/Solaris/Windows distros only RedHat (and by definition Oracle and CentOS) have the PAE enabled in their 64-bit version. I wish I knew the reason behind it.
Rodrigo Gomes wrote:And I can not give more information, because I do not see anything in the logs, the machine just freezes. And I also understand how difficult it is to reproduce this problem ... maybe we have to wait a month to see it happen again.
I do understand. In fact your setup and the amount of time it takes for something non-easily-detectable to kick in, reminded me of the famous (well, if you're a chemical engineer at least), longest running experiment in the world: the pitch drop experiment; a pitch drop every 7-13 years. Kind of tough to take any "corrective" action...

Finally, one thing I'm not still quite clear about: what is "freezing"? No keyboard/mouse input? No screen updates? No network? No HD activity? All of the above?
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Rodrigo Gomes
Posts: 29
Joined: 19. Jul 2016, 13:19

Re: CentOS 7 freezing since 8 months ago and I haven't found a solution yet

Post by Rodrigo Gomes »

socratis wrote:I was thinking more in terms of enabling VirtualBox debugging, but it wouldn't hurt doing it at the CentOS level as well
Sure, I can do this!
Can I activate the debug using these instructions?
Is there any risk in using this debugger?
socratis wrote:Excellent thinking! Take one parameter out of the equation, the host OS. Can I assume that you tried to run that CentOS physical system with the configuration that you have for the VM, i.e. replicate the CentOS VM functionality in the physical CentOS system? That would also take out the "workhorse" OS that's actually doing the job (CentOS), leaving for sure only one suspect; VirtualBox.
I actually did not try to replicate the guest machine configuration on the host machine, I just used the same operating system.
But in fact there is not much to configure, I have not changed much in the system, I use everything default basically, the only more mitigating difference is the installation of cPanel.
socratis wrote:I wouldn't personally have a clue. But if it's any help, and if I'm not mistaken, these forums, the ticket database and everything VirtualBox that you see publicly runs in a VM.
Thanks, it's good to know that you guys also use VirtualBox for servers.
socratis wrote:Finally, one thing I'm not still quite clear about: what is "freezing"? No keyboard/mouse input? No screen updates? No network? No HD activity? All of the above?
All of the above. Literally freezes everything on guest machine.
I can't even shutdown, I need to shutdown forced. I also waited some time to see if it returns to normal, but not back.
The weird thing is that CentOS does not cause any errors in the logs before the freeze, absolutely nothing. The machine just freezes and that's it.

I'm thinking of doing a update of VirtualBox, lowering the core and starting a new test.
Because I may have been wrong, something has changed since the initial freezes for now.
Previously I received memory errors and it was much more frequent.

Maybe I have had problems with different types of freezes, one caused by RAM and another by CPU. And I may not have tested it properly at the time.
But as I said before, freezes are not caused during a high use of hardware resources.
Most were caused while the machine was idle. Especially at dawn.
What do you think?
Rodrigo Gomes
Posts: 29
Joined: 19. Jul 2016, 13:19

Re: CentOS 7 freezing since 8 months ago and I haven't found a solution yet

Post by Rodrigo Gomes »

UPDATE
(After 5 months) (:

After disable "use host I/O cache" the problem has disappeared, the vm does not freeze since then.

I do not know yet the cause of this, but disabling this option solved my problem in the last 5 months.
socratis
Site Moderator
Posts: 27330
Joined: 22. Oct 2010, 11:03
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Win(*>98), Linux*, OSX>10.5
Location: Greece

Re: CentOS 7 freezing since 8 months ago and I haven't found a solution yet

Post by socratis »

Hmm... Interesting, I wonder what's causing this...
But thank you for the feedback and the efforts in discovering the problem!
Do NOT send me Personal Messages (PMs) for troubleshooting, they are simply deleted.
Do NOT reply with the "QUOTE" button, please use the "POST REPLY", at the bottom of the form.
If you obfuscate any information requested, I will obfuscate my response. These are virtual UUIDs, not real ones.
Post Reply