I gave it a try and it works. In case anyone wants to do this in the future, here's what I did to get there:
Visual Studio 2010 Express Edition
The free version doesn't natively support x64. You need to
download the Windows 7.1 SDK.
After you've installed the SDK it seemed to break normal x86 compilation using the original v100 platform toolset so I had to select the Windows7.1SDK afterwards.
I also installed Visual Studio SP1 and the
x64 compilation fix (KB2519277), as it seems Microsoft accidentally/
intentionally crippled x64 support.
Then you need to add support for an x64 target to Visual C++ (x64 is visible in the list below because I'd done it already):
Create a new platform and copy the existing Win32 settings:
Now you need to configure each of Nmap's Visual C++ projects to use the SDK instead:
There's pretty good notes (and an honest warning) available here:
http://msdn.microsoft.com/en-us/library/9yb4317s(v=vs.100).aspx
OpenSSL
The 64-bit application will need to use 64-bit OpenSSL DLLs. I grabbed the
latest version (1.0.1e) from the OpenSSL website (thankfully the sources hadn't been hacked, although the website had recently been hijacked). I also grabbed the latest version (ActivePerl-5.18.1.1800-MSWin32-x86-64int-297570) of
ActiveState Perl as you'll need it to configure some stuff before compilation. It turns out that Express edition, because it's crippled, isn't quite as straightforward as running
three or
four commands (many thanks to 3noch for doing the heavy lifting). In the end I had to add the SDK to the PATH so it could find the right binaries (ml64.exe in particular). I also ran a small batch file that, after I looked at it in more detail, called setenv appropriately (e.g. "setenv /Release /x64"), otherwise I think it defaults to x86 based on the error messages I saw.
In summary, after setting the PATH to include the right directory (e.g. "C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin" and maybe "C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\x86_amd64" although I don't appear to have added the latter), you need to run something like:
setenv /Release /x64
This should make the text green and say something like:
Setting SDK environment relative to C:\Program Files\Microsoft SDKs\Windows\v7.1\.
Targeting Windows 7 x64 Release
Then you need to run:
perl Configure VC-WIN64A
ms\do_win64a
nmake -f ms\ntdll.mak
Hopefully that works. I'm a bit hazy as to how I did it as I had to Google a lot of things after a bit of trial and error to try and get it working.
So after building the DLLs (and other files), you'll need to overwrite the files in "nmap-mswin32-aux\OpenSSL\bin", "nmap-mswin32-aux\OpenSSL\include" and "nmap-mswin32-aux\OpenSSL\lib". One thing I failed to immediately spot was that the header files in the includes directory within the OpenSSL source file are all 0KB in size! It seems that none of the OpenSSL 1.x downloads contain an includes directory with any proper files. There is, however, an inc32 directory that looks pretty similar (and seems to be what Indy Pit Crew used when creating their 0.9.8g x64 binaries). I copied those files over the top of the includes directory in nmap-mswin32-aux. I'm assuming that I compiled this okay, openssl.exe seemed to work fine.
WinPcap
You'll need to switch the WinPcap libraries for the
x64 versions you can find in their developers pack. Again, I just copied over the relevant files from the 4.1.2 pack ("WpdPack\Lib\x64\Packet.lib" and "WpdPack\Lib\x64\wpcap.lib" and copied across the latest files from "WpdPack\include" for good measure), overwriting the 32-bit versions.
Subversion
Okay, so I cheated here. I know these third party files (libsvn) are only used by nmap-update so I didn't bother doing this. For now, I just want to get the main nmap.exe (and ideally ncat and nping) working.
Zenmap
Again, I cheated here and ignored the minefield that is Python. Zenmap is the sole reason why the "vcredist2008_x86.exe" file exists in the ZIP and setup files. I find Zenmap a bit too quirky on Windows and only really use it to help generate the script arguments that I paste into the command line version. The executable is generated using py2exe, which I have little control over. If you really want a 64-bit version you can probably just run the "zenmap" Python script using the "X86-64" version of Python 2.7.6.
Compiling Nmap
Because I'd overwritten the existing x86 files with x64 versions, I didn't really change any project settings (I just had to change the platform to the Windows 7.1 SDK for every project). Also, if you haven't done this already, you'll need to make sure that you have the latest Visual C++ 2010 SP1 x64 redistributables installed so you can run what you've built. I appear to have successfully built an x64 version of Nmap 6.41SVN r31943 (I realise this dates back to August; perhaps I'll try a newer revision, although I think the very latest code might not compile properly on Windows right now).
Further Work
Obviously, the first thing is to work out why I get the horrible error message and fix that. I've basically given up for now, but I'll take another look at some point (April 2014?).
EDIT: I fixed it! In short, I hadn't included the OpenSSL DLLs alongside the executable. Once I'd copied the two DLLs across, it ran fine! This is probably my fault, as I use a batch file to compile Nmap (to avoid installing cygwin) and I think it was grabbing a copy of the DLLs from a different directory (which was empty).
The setup file still needs to be updated to include vcredist_x64.exe for x64 (instead of the x86 version) and the default location could put files in "Program Files" instead of "Program Files (x86)". It would make sense to have separate x86 and x64 installers to save people from downloading the large vcredist files (having two is bad enough, but a third version that's not even used on x86 versions is a waste of bandwidth). Nmap's WinPcap installer already supports x64 (as I got that working properly years ago) so at least that bit's simple and we can include the same file in both Nmap setup files.
For now, because I haven't updated the installer,
here's the ZIP file containing everything you need: Nmap x64 4.61SVN r31943.