virtualbox.org

4.3.12
global network setting/host only networks/VirtualBox Host-Only Network/dhcp:
default range: 192.168.56.101 - 192.168.56.254
Clear "enable server"

In the only guest, network adapter: VirtualBox Host-Only Ethernet Adapter
launch, run dhcp client:
"Lease of 192.168.56.103 obtained"

Only after closing all vbox processes, the DHCP server really stops

Did you shut down (not suspend) the guest before disabling the DHCP server? Come to think of it, "Host Only" involves drivers loaded on the host, so perhaps a reboot of the host would be useful too.

If, after investigating, you still have a repeatable bug then you should raise a BugTracker ticket for it.

Yes, I powered off the guest