SS1 mod/fan mission format

[6742edb31e882voodoo47]

ok, been talking to one of the SS EE SP devs, and we agree that a sane format for SS1 mods and fan missions needs to be put together;

Axemeleon - Today at 12:05 AM
e.g.  @3RDplayer  's mission included cybstrng.res and gerstrng.res  .... but only a few resources inside those files differ from original ones... and many of those strings are not related to the actual game, but they are used by the main menu, credits, movies.... so if we change anything in those strings to support options that were not in the original game (key binding, GPU configs etc), loading his mission will overwrite those new strings.
So my idea is that fan missions should not include those strings
Or we could establish resource numbers reserved for menus, then fan missions should always skip those numbers
That's why I'd like to hear you guys
to know what's you preference
and considering people are creating their own ports I suppose it would be nice if we had a common denominator, so fan missions could work on every port

I've suggested replicating what SS2 does, but not sure whether the SS1 res files can work in that fashion. anyway, ideas, suggestions, anything - this needs to be done right.

[6742edb31e9e5voodoo47]

and lets make sure @dertseha and @3RDplayer see this.

[6742edb31eeb0voodoo47]

looking at the current SS EE SP, the idea is to have the FM resources named differently, and write a header file that the game could load via the start a new game menu. there is a test mission available (just two files, test_mission.as and test_mission.dat), currently residing directly under \SS1EE\res\missions\, the contents of the as header file are:

/*
=============================================
Test Mission
=============================================

   Test
 
=============================================
*/

//
// Called when the mission is loaded
//
void LoadMission(void)
{
    //ResTable.SetBase("mission", "test_mission.dat");
    RestabSetBase("mission", "test_mission.dat");
   //Mission.SetArchiveFile("test_mission.dat");         // set the archive file
   SetSavePath("missions/test_mission");               // select savegame path for this mission
}

//
// Called during startup to register the mission name
//
void RegisterName(void)
{
   SetMissionName("Test mission");
}

//
// Called during mission startup (before loading level)
//
void StartMission(void)
{
   player_struct.level               = 1;      // start on medical level
   player_struct.hit_points         = 255;      //
   player_struct.cspace_time_base      = 1800;
   player_struct.accuracy            = 100;
   player_struct.energy            = 207;      // will be decremented by 208 by entry level trigger
   player_struct.shield_absorb_rate   = 0;
   player_struct.fatigue_regen         = 0;
   player_struct.fatigue_regen_base   = 100;
   player_struct.fatigue_regen_max      = 400;

   //
   // set player initial location
   //
   plr_loc_initial.x = 0x1C80;
   plr_loc_initial.y = 0x1680;
   plr_loc_initial.h = 32;
   plr_loc_initial.z = 0;                     // will be overwritten
   plr_loc_initial.p = 0;
   plr_loc_initial.b = 0;
}

this actually isn't a bad idea, as long as we move the FM resources inside another folder to keep things tidy, so we would have \SS1EE\res\missions\test_mission.as and \SS1EE\res\missions\test_mission\test_mission.dat (and change the path in the as file to  (RestabSetBase("mission", "test_mission/test_mission.dat")), meaning the missions folder would be reserved for the header files and folders with resources. there is an archive.dat file in the \SS1EE\res\missions\ folder, this is the vanilla game, so lets move it into the already existing archive folder, and change the path in archive.as accordingly (RestabSetBase("mission", "archive/archive.dat")). and do the same thing with the ReWired fan mission (renaming its archive.dat to rewired.dat - unfortunately, no idea how to rename the res files so they would get loaded properly).

and everything actually works ok as far as I can tell (the vanilla game and the test mission, anyway). so yeah, this would do for a start.

[6742edb31f3f3dertseha]

So far I don't see a reason to rename files.

Cross-posting what I just wrote to the chat:

As for the mod support, this is what v0.2.0 of HackEd was for - to get all talking and trying things out I have documented the proposed system here: https://github.com/inkyblackness/hacked/wiki/ModdingSupport
In short: HackEd stores resources of mods in files with the same names as the original. The assumption here is that mods would be stored in separate directories.
There are two cases of resources overruling previous mods (original data, in this system, is seen like any other mod):
- resource overwrites all and any previous blocks (for single-resources, that's easy, for compound resources, it means all blocks, and possibly making it "smaller")
- some compound resources are resource lists, which overwrite only on a block level (not entire resource). This is the case where compound resources contain a self-contained entity per block. Examples are string lists, such as the Trap messages.
So, with the whole thing active, the editor (or a compatible engine) would iterate over all activated mod folders, starting with the base game data, applying these overwriting rules.
When the game data is stored in "/SystemShock/data/res", mods could be stored in "/SystemShock/data/mods/...", (e.g.: "/SystemShock/data/mods/rewired", "/SystemShock/data/mods/coolTextures", ...) and in the game-menu (or startup-screen to pass exe arguments) you select the order of the mods.

[6742edb31f55cvoodoo47]

yeah, that (being able to load resources from multiple locations with priorities, and being able to load a res file that contains only the modded resources, just like SS2) would be the best.

I do think that FM resources having completely different filenames is not a bad idea, could prevent conflicts in the future. also like the idea of those AS header files.

also, the ingame menu could use some indicator to show a FM is active.

[6742edb31fcce3RDplayer]

So far I don't see a reason to rename files.

I also don't see a reason why the filenames should be different. If resources can get loaded from their specific folder, then having the same name or a different doesn't matter. It would also not matter which resources get changed. As long as they end with an empty string, they will be loaded:
The REVERSED papertext used in my mod does work now because it is changed so that it ends with an empty string.
(I will upload a separate SSEE SP beta file later today that will incorporate empty strings to resources. So that you guys can try it out.)

also like the idea of those AS header files.
also, the ingame menu could use some indicator to show a FM is active.

I also like the AS header files. Mods could be distributed with this file and the end user would just need to copy files from a zip into the missions folder and select the FM from the FM menue.

Regarding the changing of Missions. It would be very neat if the Fan Mission selector got its own submenue in the title screen. Right now its required to switch back and forth between menues to change missions, load savegames, start a new game, etc.

[6742edb31ff18voodoo47]

after working with SS2 FMs and mods for a couple of years now, I can confirm that to avoid problems when having both (graphics, for example) mods and a fan mission loaded, the new/unique fm resources are better off having new/unique names, if possible.

the Minstrel doors would be a good example - they use an edited texture with the vanilla model (as you don't want to have vonBraun doors on the Minstrel spaceship), but the model and texture filenames are the same as vanilla. this means if you have a mod that updates that texture to a higher res vonBraun texture, this will end up conflicting with the fan mission texture that is supposed to say Minstrel instead of vonBraun (because to the engine, they have the same name, so they get handled as being the same). to make sure the mod and the fan mission resources won't end up stepping on each others toes, the fan mission door model and its unique texture should have new names, so they wouldn't conflict with the already existing vanilla resources.

I'm aware that we are nowhere near dozens of mods and multiple fan missions with SS1, but if we get the framework right early, we can save ourselves a headache or two in the future.

also, I think making separate loaders for mods and fan missions is important - there needs to be a clear distinction between the two.

[6742edb3202bb3RDplayer]

I'm aware that we are nowhere near dozens of mods and multiple fan missions with SS1, but if we get the framework right early, we can save ourselves a headache or two in the future.

I agree with you and axemeleon on this one. We should come up with a standard to avoid confusion and enable compatibillity between versions and ports.

I just created a version of rewired that can be easily installed in the SP beta by use of folderstructure and .as file.
It also includes a readme with bugs I encountered while testing the changes and mission in the SP beta.

The .as file loads the archive and the citastrng but I haven't figured out how to let it load 2nd, 3rd or further resources.
I think the .as file also includes a way to turn the character, but so far even documenting it out didn't worked and the player starts facing north instead of hardcoded west.

[6742edb320462voodoo47]

yeah, I don't know how to load multiple res files either - already posted the question on discord.

have not figured out how to turn the player when the mission starts

plr_loc_initial.h = 192; should do it.

//confirming the paper working properly.

[6742edb3206e4dertseha]

I still haven't understood the problem nor its solution with different filenames.
Reason: to the engine, the filenames are not relevant. The files contain resources identified by numbers - and (in theory) it is irrelevant in which file a resource lies. It would even be perfectly possible to store all the resources of a fan-mission in archive.dat .

While this sounds like a point for having different names, I still don't see a positive effect for the user, and more negative effects for engines and editors:
* Instead of knowing just a set of filenames and configured mod-sub directories, engines would now have to have a way to specify single files (which also brings me to these AS files - I'd like to have an algorithm where these AS files are not necessary.)
* For editors, users need now to specify per resource in which file to store each (new) resource, because an editor can't automate that process (e.g. right now, all paper texts land in cybstrng.res, all audio logs in citalog.res, and so forth). I'd imagine this a bit tedious.

Even with different names, you'd still have a set of resources in one file. If you wanted to pick just one specific resource (lets say, one specific tile texture), you'd still get all the textures of that file and not just one.

Can you construct a use-case matching that of the resource system of SS1? (I don't know the resource system of SS2, and if it's filename-based, then it can't be compared).

have not figured out how to turn the player when the mission starts

Earlier today, Axemelon and I had a quick talk about this. Idea: since archive.dat already contains a zeroed-out game state, if it contains a location that is not 0/0 (which is an not recommendable tile to be in), then the engine is to consider the other location information (level, rotation) as valid and use it.

[6742edb320808dertseha]

(Note: Just because I don't see it, it doesn't mean that it is not necessary - I simply haven't been convinced for use-cases that would require a more complicated approach)

[6742edb320cd9voodoo47]

Can you construct a use-case matching that of the resource system of SS1? (I don't know the resource system of SS2, and if it's filename-based, then it can't be compared).

in SS2, you specify resource load folders in cam_mod.ini, basically location1+location2+location3 etc (with the leftmost having top priority, the second leftmost having the second highest priority etc). say you have an obj folder/zip with objects and their textures (regular files inside the obj folder/zip container) in all three locations, loc1 has obj1 and obj2, loc2 has obj2 and obj3, and loc3 has obj1, obj3, obj4 and obj5. the engine looks into the locations, loads the objects from all three, and applies priorities, so ingame, you will get obj1 and obj2 from loc1, obj3 from loc2, and obj4 and obj5 from loc3. this makes it pretty easy to load a dozen of graphics enhancing mods, and as long as the priorities are set up correctly, things will not get nuclear. and if you have a FM that adheres to the "new/unique stuff (objects in this case) needs to have new/unique names" principle, you can activate it on top of all that and it will work properly (the container names are unchanged, it's just the new objects inside that have new/unique names).

but this is not usable if SS1 stores its resources in a different format than container that includes actual files (a crf file in SS2, I'm guessing the SS1 res files would be an equivalent to that?), and right now I think that is the case, so can't pursue this direction, most likely.

basically, the question I'm asking is - if someone makes a hires (3D?) objects mod, how can we set things up so we could have this mod active in a FM, without doing insane things like merging container files?

[6742edb320f48dertseha]

Thank you for the explanation - and I believe we're on the same track here. The layering I envision (and currently have in HackEd) works the same: a list of mods to work through in order. resources of files from "higher" ones override those below (to clarify: on a resource - possibly block - level, not file-level!)

Let me take your example of a high-res 3D objects mod. Their geometry is stored in obj3d.res and (if memory serves right) their textures in objart3.res (even if not, let's suppose that's the case).
Vanilla game has three objects, Obj1 (resource ID 1), Obj2 (resource ID 2), and Obj3 (resource ID 3), geometry stored in obj3d.res, with textures for them in objart3.res

A high-res mod is created, where the author could only make Obj1 and Obj3 (because, for example, Obj2 was already good enough - There's only so many polygons you can have in a microwave oven
This mod would have a obj3d.res in folder "highres3D", with the respective resources being stored: Obj1 (with resource ID 1) and Obj3 (with resource ID3).

The engine then would load, based on the priorities, first the vanilla obj3d.res, then the "highres3D/obj3d.res". In the end, Obj1 and Obj3 would be taken from the highres mod, Obj2 still from vanilla game.
The concept "new stuff with new IDs" can be applied here as well: "fancy-fm/obj3d.res" provides just one object, Obj4 with ID 4 - and the "fancy-fm/archive.dat" can reference this extra model, as well as the Objects 1-3. In either the high-res variant or without (if highres3D mod was not activated).

There could be an independent high-res-textures mod, making fine-detail textures for 3D objects (stored in objart3.res). Assuming this Obj4 from fancy-fm was referencing the vanilla textures (didn't need extra textures), it would automatically benefit from this high-res-textures mod as well.

[6742edb32107avoodoo47]

would this allow multiple obj mods being active/loaded at the same time? because I think we need to be able to load multiple container files of the same type (obj3d.res in this case) at the same time for this to make sense - say we'll have two object mods, one improving soda cans, the other one beakers, and we'll want to load them both without having to merge the containers.

[6742edb32124fdertseha]

yep, that's possible (in my idea):
you have mods "sodacans", "beakers", and "microwaves". each of them comes with their own obj3d.res, which only contains the one respective object.
Then, somehow, the mod-order would be "mods/sodacans/obj3d.res", "mods/beakers/obj3d.res", "mods/microwaves/obj3d.res", "data/res/obj3d.res" (going from left to right here, left takes precedence; "data/res" is the vanilla folder).

If you put it that way (one resource per mod), you already have kind of a resource-specific filename. It just includes the name of the mod with the subfolder

Yet, Axemelon (with whom I'm in chat with at this moment as well) pointed out some edge cases and possible problems with (at least) languages, and the demand to change languages at runtime. Though, I just brought Axemelon to a point where they think about my idea - I'm not sure whether I confused them to muddle their use-case, or I haven't fully grasped the limits of the engine (work-time to modify the engine is probably also limited).

There's a bunch of factors at play: limits of the engine, allotted time of Axemelon, workload of future users, and my possible re-rewrite of the new editor if things change.

[6742edb32138avoodoo47]

yeah, no system is perfect, most that work are just good enough - every once in a while, I manage to find some mod/fm in the SS2 ecosystem that does things differently and does not work properly as a result (and must have its innards rearranged or something similar - the Minstrel FM would be a good example, that one required some serious effort to get going right).

good thing is that we won't have to take legacy mods into consideration (as pretty much none exist), so if we do this right, and all the future modders will design their stuff according to the final specs, we could have an unusually smooth ride ahead of us.

[6742edb3217ceZylonBane]

the engine looks into the locations, loads the objects from all three, and applies priorities...

Thanks to ProcMon, I can say for a fact that the Dark Engine resource loader does not work that inefficiently. What it does for every resource request is search for the resource in the highest-priority path, then the second-highest, etc, until it finds it or fails. When it finds it, it loads it and stops searching.

If SS1's resource loader could be expanded to handle "bare" assets not wrapped in RES files, that would certainly make modding a lot easier.

[6742edb3219f7JosiahJack]

I agree that loading an entire res file unnecessarily, only to have a huge percentage of it overridden by the next res file would be a pointless and sluggish operation.  I don't know if the res load operation can be broken down to only load indexed resources that haven't been flagged as loaded yet (bool array?). I do like dertseha's method.  Floating file assets leads to some extra parsing headaches such as what filetypes do you support, licensing headaches, etc.  But maybe someday!  My 2cents on filetypes: .png, .obj, .txt, .wav, .midi

Seems like there needs to be a modorder.ini file. Something that specifies the pecking order:
1:modnameoffolder1
2:someotherfolderforamod
3:joecoolsodamod
4:3RDplayer_fm2

etc. to signal which mod to parse first. This list could then be either sorted manually (blech) or added to a page in an in-game menu of mods with up and down arrows to sort the list and toggles to disable/enable loading that mod.

[6742edb321b8bGawain]

All assets in system shock are so small that overriding them (even like 100 times) shouldn't be any problem.
You actually don't have to load everything, just parse the headers a little and find out what files have what.

[6742edb321e4bJDoran]

When a mission loader is created for System Shock, whether it's a separate program, or compiled as part of the game's startup .exe file, or whatever, then I think it would be great to have the following features:

1. Different save files for each mod and the plain game. That way, whenever the player starts up either the normal System Shock, or System Shock with a mod or fan-made mission, the full save bank is available to be used for that play session and any prior saves already made with that same setup up (such as System Shock and a particular mod) are available to the player automatically.


2. Will any mod or fan-mission ever require extra options or configurations that the base game doesn't (and can't easily support)? Such as extra keys (keyboard keys, I mean) used for in-game functions that are new to a certain mission? For example, if a fan-mission introduces an upgrade for the player that allows him to slow down time, 'bullet-time' style, by (once he's found and collected the upgrade, in game) pressing 'T', or whatever key the player chooses to define to activate the function?

If so, then it would be great if the mission loader could handle the .cfg or .ini files automatically.


3. A text viewer in the mission loader would be great, so that there'd be a button in the UI that loads and displays the readme.txt, provided of course there is one in the fan-mission/mod's directory or archive file.


4. It would be nice if fan-missions and mods came with a screenshot showing the mod, and when the mod/fan-mission was highlighted in the mission loader then the screenshot would be displayed.


5. An option in the mission loader to disable the game's splash screens and opening FMV would be nice, I think.

[6742edb321fc5voodoo47]

1. is already there and working.

also, we need to make a clear distinction between fan missions and mods - mods do not need separate save folders, why would you need extra saves if you enable hires sprites mod, for example. so mods - stuff that updates vanilla resources, fan missions - new levels you can play. what currently is available in the SP beta is technically a fan mission loader - I'm hoping for a separate mechanism/tool that would load mods in the future.

as mentioned, the current SS2 system is a good example of how this should work - basically, you activate the desired (graphics, audio) mods, and then you either run the vanilla game, or a fan mission, and all the mods will be active in both cases.

not a big deal at the moment, as the number of audio/graphics mods for SS1 is pretty much zero, but it's definitely something we want to have working in the future.

[6742edb3220edJosiahJack]

There are 2 different types of mods:
Mods that enhance specific features and are intended to be layered/layerable.
Mods that make significant changes to gameplay, features, AND include levels.

Then there are maps, aka fan missions since SS lends itself better to encapsulated stories rather than single levels since there isn't technically an "exit" to levels (yet?).  Edit: by exit I mean a simple way to quit without requiring fighting SHODAN in cyverspace.  E.g. an elevator button.

[6742edb3221e3voodoo47]

to make it simple, modified/enhanced/replacement stuff for vanilla resources = mod, new stuff = fan mission.

[6742edb32232eicemann]

First RPS, now Kotaku