I've spent the last day or so trying to get Solaris 10 GA working on VirtualBox. It's not that easy
thanks to this bug:
Solaris 10 was released with a somewhat schizophrenic video subsystem
...
For S10U1, an easy option is installing using the text mode installer from a console session, choosing the "Solaris Interactive Text (Console session)" selection early in the installer. That unfortunately does not work with S10 GA which will execute kdmconfig anyway, crash, and cause the console to be all black
But I think I've managed to work around it by forcing the output to ttya and having VirtualBox create a named pipe then connecting with putty to do a text based installation. Files are copying as we speak. I hope, as I'm now at lunch.
EDIT: It actually worked. I got Solaris 10 x86 GA working in VirtualBox. While I tried to configure a few things, I hit a few Solaris quirks (such as logindmux screwing up, which stopped telnet working), but I managed to fix them too (by removing then re-adding it and rebooting the system with the reconfigure flag). I tried creating an xorg.conf file to run everything at a lower resolution, but that screwed up so I abandoned that idea for now.
In case anyone else wants to try it, I configured COM1 in VirtualBox to output to the named pipe \\.\pipe\solaris (you can call the named pipe anything you like, e.g. \\.\pipe\foo). I then configured Putty to connect using a serial connection to \\.\pipe\solaris. When Solaris 10 boots the ISO image you have a few seconds to hit the Escape key. This brings you to a text based interface where you can use the function keys to set the output to ttya (I'll try and add more details to this bit when I actually have the VM in front of me). You should then see the text based interface in Putty. Stick with that for the rest of the installation (when it asks, tell it you're using xterm, as that's what Putty emulates), don't switch back until everything's done.
One final thing, some people have written that not allocating enough RAM to the VM can cause installation problems. I went for 2GB and didn't have that problem, but it sounds like you may need to allocate more than 1GB despite VirtualBox suggesting less.
