It's very much possible to install Windows NT 3.51 in VirtualBox (and without needing the three boot floppies, or being limited to VGA resolution).
Here's what you need (in addition to your NT 3.51 CD or ISO image):
1. An MS-DOS 6.x boot floppy image (with CD-ROM support); a 6.22 boot floppy image works best, naturally.
2. The UNIVBE VESA driver for Windows NT 3.x (a Yahoo or Google search should turn this up).
With the images mounted in the appropriate places (and a 1 GB hard drive image created), boot from the floppy image, then FDISK and FORMAT the HD partition, rebooting after each step. (Do NOT copy any files to the hard drive image!) After the post-FORMAT reboot, change to the \I386 directory of the CD-ROM drive image. Then type:
winnt /b
The /b switch copies the setup files to the hard drive (and eliminates the need for either the CD image or boot floppies once the copy is done; in fact. NT Setup will prompt you to remove the floppy drive). Unmount both images and restart your VM. Windows NT Setup's second and third stages will start normally; follow the prompts as you normally would, sticking with VGA resolution. Once this is done, and after the post-stage three reboot, re-mount the NT CD or ISO; there is a file in the \I386 directory you need: FRAMEBUF.DL_. From the Command Prompt EXPAND X:\I386\FRAMEBUF.DL_ to C:\WINNT35\FRAMEBUF.DLL. (By placing the EXPANDed file here, you won't have to worry about it later.) Once that's done, install the UNIVBE driver via Display in Control Panel. (When you do that, you will get an error that it can't find FRAMEBUF.DLL; ignore it, as your EXPAND took care of that.) Restart the VM. Now, you should have 640x480 in 256 colors. Go back to Display in Control Panel and adjust to suit. (I chose 1024x768 in True Color.) Once that checks out, and you reboot again with your new deeper-color desktop, you can install your favorite utilities (IE, etc.) and Service Pack 5 (the highest available for NT 3.51).
Hope this helps!