Solved. If you are encountering the same problem (which means your system will
*NOT* prompt you to allow loading system extensions from the preference pane), the solution could be to do a --clear-staging:
The above deleted few .kext inside /Library/StagedExtensions/..., probably leftover of previous old installations. Somehow, files in there prevented the system to prompt the user to "allow" Oracle in Preferences -> Security & Privacy -> General.
Tested and confirmed that now all my VMs are running.
So here it is in details my cleanup process:
1. Uninstall VirtualBox with the uninstall tool found in the dmg
2. Restore SIP to default: sudo csrutil clear
3. Reboot
4. Clear staging: sudo kextcache --clear-staging
5. Reboot
6. Install VirtualBox and follow system prompt to allow Oracle