Tool for converting Ultima Underworld and Shock 1 Levels into Unity.

[674098001efa4hank morgan]

Realised I haven't posted much lately. Just dropping by to say that I'm still working on the tool. I took a brief holiday and I decided to just go dark for a couple of months. I hope interested followers of the project will be patient as I'm just going to keep things under wraps for a while. I am hopeful for the future.

[674098001f1f8icemann]

Personally I prefer the occasional updates, as it's quite fascinating reading about what goes on with these types of projects. Even when it's just updates about dead end attempt at getting things working, as the attempt in itself with the steps taken is the good reading part.

[674098001f374hank morgan]

One little story. For a while last weekend I was being chased around a UW level by a stack of books. I'm still not sure why it happened but they eventually trapped me  in a corner.

[674098001f4f1Kolya]

Haunting and educational, that's how I like anecdotes.

[674098001f5f4OmegaDEATH]

Interestingly... several years later fans have forged and path. : )
I'm so proud. 

[674098001f784WhyNott]

With Source exporter, it would be possible to convert SS1 levles into Garrysmod maps, which would be pretty awesome by itself, actually. Also, a level editor would be phenomenal since as far as I know no one ever got even remotely this far in reverse engineering System Shock.

It's kind of sad that games like Doom or Wolfenstein 3D got fan made editors weeks after release, while after 20 years we still yet have to reach that point. Matter of popularity, I guess. 

[674098001fc05hank morgan]

Ah what the hell. I may as well just let the cat out of the bag. It being the 20th anniversary and all.  So anyway after I did the whole source engine export I spent a small bit of time exploring what I could do with that engine. It's a lot of fun loading a UW level up in Dark Messiah and just kick orcs off of high ledges or into spike traps.

All this lead me to a program called crafty which can convert Source map files into .obj files. The Dark Radiant inbuilt .obj exporter never really worked for me so it was nice to get the level map out as a model since interested parties could just use it with whatever engine they want. On a whim I had also installed Unity3d for the kicks and imported the model into unity. Sadly there were a whole load of missing texture faces however if I then took that obj file and passed it through Autodesk FBX converter I'd get a file that works. On foot of that I added a fbx converter (using the Autodesk SDK) to my original program which does a full map export that is almost perfect except that I've yet to figure out how to align textures. By that stage my brain had truly melted. Regardless I could still manually take a source engine export, turn it into an .obj and then into an .fbx and bring it into unity and have correct textures or do it directly with un-tiled textures.

So that just left me with a model in unity with nothing in it. Pretty useless right? As it turns out unity is pretty flexible in allowing you to extend it's editing tools so that I could create a custom menu that calls a special script that can populate a level with game objects and attach whatever properties I want. So on foot of that I added a bit of code to my tool to dump a list of every game object for a level into a simple format which calls creates a uniquely named game object, position it and then attach a default sprite to it in a separate function.

Code: [Select]

myObj = new GameObject("a_goblin_36_28_00_0225");
pos = new Vector3(43.714287f, 3.853347f, 34.114285f);
myObj.transform.position = pos;
CreateObjectGraphics(myObj,"Assets/Sprites/objects_077.tga");

So as long as I've imported my graphics assets(which I can generate from my tool) I can instantly populate a level with the basic objects. As I expand my implementation I can just insert further properties be they object scripting, collision meshes, nav-agents, animations, or whatever. As well as all that I can create a script that renames the tile meshes so I can interact with them easily in code for implementing elevators and the like.

I'm still tooling around with various things. There is obviously a bit of a learning curve here, I'm doing a lot by hand while I prototype various things and in most ways I'm back at square one but I'm pretty excited by the possibilities. I think problems I had no grasp on like inventory systems, interface, persistence are now solvable problems to me. I don't have much to show off at this point but here is the thing I'm most proud of. A rudimentary AI. A (floating) green goblin from UW. He has 8 different idle animations depending on your angle of incidence and he can navigate around the navmesh of the map.

Image: http://i.imgur.com/29fZlsq.jpg

For those of you not familiar with the guts of my tool then I'll just point out that I conform Underworld and System Shock tile-maps and object lists into the same format. So although certain behaviours are different the basic output of data is the same. So I could just as easily be showing you a populated SS1 level with a med-bot scratching his head.

[674098001fd34Carrot]

Very cool Hank!  It seem to be a little scattershot with all these different engines (and how many convertions to get from SS1 to Unity?) but it's not like you have a schedule you're trying to stick to.  Have fun and explore.  The options in Unity sound very promising.

[674098001ff58icemann]

Very interesting. Your nearing the maximum that TSSHP reached over a decade ago. Though they never got an AI to actually move. They'd just started on that side of things when all progress stopped, if I remember right.

As soon as you get audio log playback, music playback and rudimentary display of all the enemies (with them just showing their idle animations), you'll be about dead level to where they got up to. After that you'll be treading on ground never reached before.

One question. Will your efforts for now be towards AI behaviors, or for more player oriented things like user interface, inventory system, ability to pickup items, implementation of all the various items and weapons (plus the ability to use them, which would require the inventory and user interface systems to be in place first) as well as player activated triggers in levels (ie switches, force bridges, level events like elevators dropping after the player walks over a trigger or does something etc etc) ?

Any of those would be quite time consuming to implement I'd be betting.

[67409800204b4hank morgan]

My (UW focused) rough plan is as follows. I'm still at the baby steps level so I'm dipping my toes into a few topics randomly to get an idea of how everything works.

Implement an extremely basic hud for just showing debug output of scripting and interaction messages. I've done mock ups of the UW interface using a gui plugin just to get a handle on things like view-ports and layout. Allow the ability to switch between mouse look and free mouse interaction.
Finish out the map export to include door frames and any other static features.
Working doors and keys.
Buttons, keypads and regular switches.
Get back to the point where I was with level event scripting (except for lighting), i.e. elevators, message barks, changes in terrain, timers, teleports etc. I'm hopeful that this won't take long given the capabilities of the scripting system since I can just instantiate the various script types on the traps and triggers as needed with the correct parameters.
Reimplement decorative level objects such as gratings, tmaps, signs, screens etc.
Very basic pickup of objects and inventory display. Simple inventory interactions like activate and examine. I've already played around with a free inventory plugin but it didn't quite meet my needs. Along with this I still have to match up the SS1 objects correctly with the right artwork.
More advanced inventory. Ability to drop and throw objects, equip weapons and armour.
Implement 2 weapons templates. Close range and ranged.
Complete the full animation set for the goblin. I won't get deep into other animations until I have a good process for this. Eg for UW it's about 64 different creature variations due to colour changes so I'm talking about 8 walk, 8 idle, 3 combat  and 1 death animation each. I think I should be able to automate this process but I've yet to figure it out.
Basic AI. Wander around randomly within an area.
Basic AI. Peaceful and hostile states. Simple attack AI.

Beyond that I'm looking at expanding UI stuff, improved AI, triggering conversations, level transitions, revisiting lighting, music, my first magic spells, stats.

Once I get up to the point where I have some level scripting working I will release a demo.

Very cool Hank!  It seem to be a little scattershot with all these different engines (and how many convertions to get from SS1 to Unity?) but it's not like you have a schedule you're trying to stick to.  Have fun and explore.  The options in Unity sound very promising.

The number of conversions(Source->obj->Fbx) is a bit much at the moment. Like I said I do have an unfinished FBX exporter of my own that I will polish off some day. I think I'm done with new engines now.

[67409800205c1WhyNott]

Are you going to release the Source importer anytime soon? It could be pretty useful for me, since I'm really into source mapping lately.

[6740980020709hank morgan]

It's already out. You'll just have to get your hands dirty and compile yourself.

All my released code is at https://github.com/hankmorgan/UnderworldExporter

[6740980020879WhyNott]

Oh. I didn't notice, sorry.

I'm a total noob when it comes to programming. I tried compiling your exporter, but I ended up failing due to errors. I used Visual Studio 2010 and it gave me compile errors like "warning C4101: 'bab' : unreferenced local variable", "warning C4244: '=' : conversion from 'long' to 'float', possible loss of data", and " warning C4101: 'auxPalIndex' : unreferenced local variable".

I really don't know what to do. I know that it's probably an error on my side, but as I told you I have little experience with coding at all. I know that asking you to compile the thing for me would probably be too much, but if you could just hint me as to what am I doing wrong.

[67409800209efKolya]

I have no idea about VS 2010, but what you posted are warnings, not errors. And none of them sound very severe (variables declared but not used and a data conversion).
Were there any actual errors?

[6740980020bd9hank morgan]

Most likely you are having problems since the Autodesk FBX SDK http://www.autodesk.com/products/fbx/overview  is part of the most recent release. I think this following link should give you the previous commit before that requirement. It has no dependencies. Once you get it working just make sure the various config files in the program directory point to your original game paths. It probably easier to try and run that version. You'll need to extract your textures as well. Just make sure the texture file name are the same format as in the matching texture config file.

https://github.com/hankmorgan/UnderworldExporter/archive/552c649618908b59b17ee4240699d5b012a96e5d.zip

I'm a lousy programmer who just has a lot of perseverance so I'm not sure if I can be of much help here in terms of setting up your environment. I'm coding on VS2013 but I've also used VS2008 as well but if you can get to a specific set of errors then maybe I can see what the problem may be.

[6740980020d09hank morgan]

Actually I do have a build on hand on this pc. See attached. I'm not sure if you'll still need to install the sdk but I've included a dll with the zip.

Use at your own risk.

[UW exporter release.zip expired]

[6740980020e44WhyNott]

Thanks a lot! That exporter executable works like a charm. Now I only have to convert a few hundred wall textures into vtf format, but that's not a big problem for me. The bigger problem would be to figure out which texture is which since the vmf file that your exporter produces gives them somewhat enigmatic names like "shock_226", but I'm gonna figure that out eventually. Once again, thanks!

PS. What does the "Repacker" mode do?

[6740980020f59WhyNott]

Oh, I just figured out that shock_texture_config_source.txt explains which texture is which. Disregard what I said about the textures. Great job!

[67409800210f6hank morgan]

VTFEdit should allow you to batch convert all your textures.

Repacker mode is for recompiling UW2 and SS1 datafiles. Basically those data files come in a partially compressed format but the repacker rewrites the data and the internal archive indices so that those files can be stored uncompressed yet still usable in the original game. I use it for object research when I want to see what changing a value does in the original.  Eg you can repack SS1 ARCHIVE.DAT into an uncompressed file and then properly edit object properties in a hex editor.  Just back up your original files first if you try it.

The texture config files (eg shock_texture_config_source.txt) will tell you what each texture file should actually be named in order to be picked up properly.  The number is just the internal texture number. Just keep the numbers in order if you have to rename while converting over to vtf and everything should work out.

[6740980021238WhyNott]

Well, I've just managed to get everything to run!~There are still a few textures missing for some reason, but the whole thing even manages to compile without a leak.

I suppose that the next logical step for you would be to add an option to export lights into source. I recall previous exporters (namely R3D) having that feature, and since you even managed to get the doors to work in your DarkMod exporter, it shouldn't be a big deal for you.

Well done, and thanks!

[67409800213fahank morgan]

I think you are my first actual "customer". Enjoy.

On my end I'm just prototyping little bits. I'm using UW1 as my test platform at the moment but I have a familiar little HUD mocked up as an example. I'm viewing the game-world through a viewport with some buttons on the left. I can switch between mouse-look and mouse interaction at a key press and can interact with the switch in front of me in both modes as well as trigger events in the hud area. The button is linked to a do trap object which is spitting out a debug message to the scroll area.

Image: http://i.imgur.com/XJgy44f.jpg

The white borders around the art is probably due to alpha channels on my exported graphics. Something to fix.

[674098002196ficemann]

Complete the full animation set for the goblin. I won't get deep into other animations until I have a good process for this. Eg for UW it's about 64 different creature variations due to colour changes so I'm talking about 8 walk, 8 idle, 3 combat  and 1 death animation each. I think I should be able to automate this process but I've yet to figure it out.

My experience in this area is more in the 2D tilemap platforming direction with remaking old console games, but with my last project I wasn't able to fully automate that part of it since several of the enemies had different amounts of sprite frames. Quite simple to code together for each individual one in either case. Just very time consuming. Simple for loops to cycle frames did the trick quite nicely.

Same with attack animation loops etc, with checks for once frame x was reached to perform x action etc.

[6740980021c36hank morgan]

So I've been spending a bit of time on UW's script system. Before when I did UW scripting in IdTech I just followed what I called the script chains : Activator -> trigger -> trap -> further triggers -> further traps etc etc in a linear fashion. So for example the function I generated for a switch would also contain all the code for each trigger/trap that was fired by it. It was sufficient for my purposes at the time. When I did SS1 scripting in IdTech I went slightly differently and for each trigger I just used a trigger object that would carry out it's own scripting again it was what worked and was close to vanilla game behaviour and was a much simpler system to understand and debug.

So now in Unity it's not feasible for me to do scripting in the old UW fashion so I've had to automatically generate trigger and trap objects as their own predefined script triggers that just do their own action and in turn fire off any other trigger/traps that are need. Just like in the early nineties.  So far for most objects I just get back debug printouts saying what has been triggered so I can inspect the "chain" and it looks okay so far. The only fully functional script object I've got at the moment is the elevator object which is significant because it also proves that I can manipulate the map model  a runtime.

All this means I now have working switches and other clickable activators that when I activate them will fire off other objects. From there on out it's a matter of filling out those script object templates to perform their correct actions with the right parameters.

This system of activation should be able to carry on to SS1 without much head scratching.

I've archived my current unity scripts to the project github for anyone is interested in the process of generating the gameobjects automatically.
https://github.com/hankmorgan/UnderworldExporter/tree/master/UnityScripts

[6740980021e41hank morgan]

Here's a prototype of Level 1 of UW1.
http://www.mediafire.com/download/ydossqxs2qnbm9e/test.zip

Controls
Standard fps but press E to switch your mouse cursor mode. Make sure you set your graphics level high as low settings look like crap.

All very rough.

Known issues.
Object scaling is not done and objects are floating off the ground.
Player movement is not yet tweaked so it's very fast and floaty.
AI's have no collision meshes.
All AI's are goblins.
AI's will follow you when you click on them in attack mode. This is deliberate for showing pathfinding.
Rapidly clicking switches may have unintended effects on their targets.