SS1src: Pre-Planning with the source releases

[674224c62cad1dertseha]

°SS1src

I believe it is time to consider what to actually do with the source release - regardless of whether we'll base any work on the Mac-release, or, more likely, on the source port that NDS promised. We've been waiting a long time for this and we should be prepared for when it actually happens. By this I mean to have a plan and strategy on how to work on the source to make it do whatever people dream of. Like, lens-flares, selfie-stick-screenshots, and automatic Instagram-status-updates on plot events. You get the idea. Because to get there, much needs to be done.

I have never worked on a large-scale open-source project with several people so far, which even has the added property of being legacy code by both of my known definitions. So, for my ideas I will rely on my day-job experience from working on safety-critical software, influenced & supported by agile processes and XP practices. I like to plan a little bit ahead, and also have an eye on quality. While this doesn't always apply to my private hobby projects, an undertaking such as working on a product with a legacy on its own like this one requires more care. You will hopefully agree on that.

While it is fun to dream about all the big features that a future engine would support, the base needs to support that - and here's where I have the most ideas for now. With my sketch I'm inviting you, all current and future devs alike, to chip in and work with this vision, both by contribution, criticism, and support.
For now this should be a high-level perspective and we want to avoid bike-shed discussions. At this point I'd rather like to form a cohesive community focusing on one engine, rather than seed the thoughts of early forks already.

[674224c62cd0bdertseha]

Phase 0: Get a grip on the material: documentation

This is all about making the code itself accessible, which is already happening right now - the mac source can be a basis for this already. So I guess this phase is a parallel one to all others. It can also influence other phases.

Including:

• Filling out the cracks of ss-specs about all the resource files: This allows tool makers to work against a spec, rather than wading through sources. This also helps future engines for compatibility reference.
• A separate document, describing the engine itself. This is a two-parter: One about component diagrams and data flows from a high-level. From these you can learn the current code, and plan ahead for feature extensions. The second part is the (public) function-level documentation. Both can be based on Doxygen, with a PlantUML plugin for the diagrams, so that it is always close to the source.

If this Doxygen document is even integrated into the build chain (see below), then you'd have a current documentation each build.

[674224c62d012dertseha]

Phase 1: A reproducible, common chain: from writing code to binaries

This is all about making the contribution-chain accessible to anyone interested, with as little friction as possible.
With this I mean things like

• A reproducible build chain. Ideally this would be a docker-based build that people can use to generate the binaries (at least for Windows).
• A "common" build system. Assuming we'll stay in the C or C++ world, CMake would be my first pick. This also allows people to work in their IDEs as it supports creating project files for most common ones - including Visual Studio on Windows.
• A defined set of styleguide and naming guidelines (rules). This includes a configuration file for an auto-source-format that is required to be adhered to. I'd rather have the fight once for the rules in the beginning rather than regularly at each pull-request. Shared code-ownership requires to give a little in order to enjoy many benefits. (As they say about the language-defined formatting rules of Go: "Everybody hates them, and everybody loves that they exist.")
• A unit-test system prepared, and hooked up into the build chain. I'm voting for google-test .
• Possibly a decision on compiler and version. I'd like to avoid chasing fancy new STL features, hardly supported by all compilers. My vote goes for C++14.
• Documentation (!) on all of these elements. Even if this is all automated, it should be documented how it works, and why.
• Some sort of task tracker. Could be Trello, or the new features of GitHub.

With all of that, people can come in, clone the repo, set up their IDE, code whatever they like, and then cross-check with the automated build-system before making the pull-request. Any everyone's contribution will look & work the same.

This phase shouldn't require any drastic code-changes of the base material (I'm not counting the one happy-day commit of the reformatting) - and not even thinking about any "quick fixes" or "fun features".
Even if we can't agree on all the later things, at least these fictional forks will have it easier.

[674224c62d1federtseha]

Phase 2: Cleanup & Compatibility

This is about driving the code into a direction that supports new feature ideas (where it doesn't already).

This can include:

• Multiple platform support (if the NDS release doesn't already)
• Defining a compatibility mode that lets the new engine load old resources, and activate new features on new ones
• Separating game-logic from engine-logic: there are many object-specific hacks in the source. See if these can be extracted into data-driven behaviour for mods. The first mod is the original game.
• Finding component seams for adding first tests
• Removing any unused stuff, such as leftovers from other (unknown) uses of the engine.
• centralize constants and/or make them data-driven

This again might be something that runs parallel. Yet, the more that is completed before the big features, the better.

[674224c62d33bdertseha]

Phase 3: Big ideas

Now we can start to think big. Which features should it support? How to modularize the system for best efficiency? and so forth.
This is where my ideas become sketchy, because for one this is months away and depends heavily on public demand.

At least I'd see it relevant to modularize the engine through interfaces. Perhaps it is this phase where the bigger refactorings will happen. In my ideal world, we'd be able to have automated tests for the various components via their interfaces and a good understanding of the various dependencies.

[674224c62d428dertseha]

...aaand opening the floor for all.

Thoughts?

 

[674224c62d5d6icemann]

I'd recommend that phase 1 be getting the code to compile in Windows natively (via Visual Studio preferably) and for it to run natively in Windows without the need for a Mac emulator.

Best chances of source ports later on down the line that way. And it will reach a FAR higher user base.

I'd also recommend setting up some kind of todo list. Easier for multiple people to work on it that way, with different people assigned to various tasks/features to get working / implement.

Note - This topic is set to only allow admins and moderators to reply currently. So if you want to get replies from everyone that will need to be changed.

[674224c62db75dertseha]

I'd recommend that phase 1 be getting the code to compile in Windows natively (via Visual Studio preferably) and for it to run natively in Windows without the need for a Mac emulator.
Best chances of source ports later on down the line that way. And it will reach a FAR higher user base.

Yes, the Windows build has preference. I am assuming this is what the future NDS source port release already provides out of the box. I'll add a clarification above.

Note - This topic is set to only allow admins and moderators to reply currently.

gah - so many buttons! thanks for the heads up - I thought I had unlocked it.

[edit]
also thank you for the hint on the task-list. Another thing I assumed and didn't explicitly mention. Added to phase 1.

[674224c62dda9voodoo47]

Phase 2: a 3D accelerated renderer would be a good start, as that means proper resolutions, fov and rendering speed on current computers. proper/modernized controls are also necessary, either EE style or something better (I suppose the code can be ported from the last public SHLink release, if necessary).

once we have a windows port with d3d/opengl (or vulcan, in case someone would want to go nuts) and full mouselook support, we can think about the potential fancy stuff. btw, it would be very good to know whether the NDS port supports hw rendering (the assumption is it does, as a sw renderer is not a very good idea nowadays, but would be nice to know for sure. I'll try to ask their guy at steam). //it does, opengl.

[674224c62e159dertseha]

once we have a windows port with d3d/opengl (or vulcan, in case someone would want to go nuts) and full mouselook support, we can think about the potential fancy stuff. btw, it would be very good to know whether the NDS port supports hw rendering (the assumption is it does, as a sw renderer is not a very good idea nowadays, but would be nice to know for sure. I'l try to ask their guy at steam).

It is still my optimistic impression that NDS is already doing the OS/graphics abstraction. Perhaps they are already basing this source port on SDL (or glfw3) for instance, and for us it's a walk in the park then. For me it would be a bummer if they used D3D directly and we'd then have to rip it out again to get OS-independence. My first and primary concern is to get OS-independence, and then do HW-acceleration.

If this doesn't hold, and its still some Windows-specific software renderer, then I don't know whether this port is possible in phase 2 already. I still have no idea how to do a 3D renderer and from that perspective, I'd first like to have everything properly componentized via defined interfaces - at least the renderer. So, I'm more reserved on this point, though someone with better knowledge on renderer and gaming engines can surely and happily trump me on that.

[674224c62e269voodoo47]

it would be really odd for them to not do a modern renderer (my guess is that modern resolutions, proper fov and 30/60FPS are impossible otherwise), but who knows.

[674224c62e3c7icemann]

SDL does things a million times better than directX / direct3D ever did. I'm currently teaching a direct x unit. After working through SDL on CrossTrix for so long. It takes a fair bit of code in directX what you could do in a few in SDL. Give me SDL anyday.

[674224c62e577dertseha]

Semi-related to this planning: I was just offered on the Discord server of NDS to open a channel for source discussions.
I'm hesitant to accept, so I'm asking for opinions.

Personally I'm leaning towards rejecting this offer (though, NDS can still decide later on what they want on their servers...) because to the only benefit of having the ability of instant-communication, I see the following drawbacks:
* not public, and not all from here are registered there
* not a public community server
* Fracturing of the communication channels at this early point (I'd rather think about having the chat on GitHub, which they support now, or simply stay on these forums here) - right now we want to focus people on one point
* These forums here have better documentation (archiving) quality.

Yes/No/Don't care?

[674224c62e8c3Bert3]

the first steps in my opinion:

-get the current development environment running: this is the only (working) reference available

clean full description how to install/compile/run shockmac

maybe someone can provide a complete image based on
https://www.macintoshrepository.org/ (MacOS 9, CW 10 Gold Tools (IDE 1.7), System Shock for Mac)
or http://macintoshgarden.org/

for SheepShaver or the latest qemu seem to start supporting MacOS 9 good enough to be also useable
->this part can be done by anyone - even non programmers to support programmers who just don't want
to fiddle with the installation problems

-get all the included Library test projects running (helpfull for porting)

-get a simple unit-test for integer size of short/int/long/... on the reference platform - sizes could be sometimes strange

-replace the PPC asm with C - while testing on original platform
or some can use qemu and Debian 8.10 for testing the port: https://gist.github.com/nstarke/c6593da117c459c18c255395211fa311

-can check if PC/DOS data is different to MAC data

-understand how the Mac game data needs to get changed due to the big endian nature of PPC, major work can be needed if the data format is not endian aware - for example: direct reading/casting of integers

- start to think about porting to other platform, this is the time when CR+LF gets interessting etc......, and yes CMake would be great

keep the communication here: the forum works and is well known, and some github fork will become the master over time - that is nothing which can be decide beforehand - without having already serveral developers working together on the port

[674224c62ec2cvoodoo47]

I was just offered on the Discord server of NDS to open a channel for source discussions.

kind of depends on what they have in mind - if they maybe want to cooperate with us (or vice versa) at some level and want a place where our devs could talk to their devs anytime, then it might not be a bad idea. as mentioned, we want to avoid parallel development and/or stepping on each other's toes if it can be helped.

registered on discord btw, still looking around finding out what's what.

[674224c62eda9dertseha]

I have forked the mac build discussion into a separate topic. Awesome stuff!

As for the topic at hand, I still hope that Stephen comes around here soon ( he told me in private communication he would come here) to discuss the second source release. Because as fun as it migt be, if this second source release is essentially around the corner - within the nexr weeks - then I don't see a point in trying to port the ported code back a second time.

*) Stephen wanted to talk with me about the second source release several times, though with our time shift, that sometimes fails. Still, from what I've gathered from all his messages, I get the impression that we are way better off waiting on this second release.

[674224c62f0f1dertseha]

Ah, my posts were moved to Getting the mac source to build reproducably.  Thought they were removed completely.

Heh, I'm trying to avoid being a dictating ruler here what's actually the case is that I'm new to having moderator rights and still learning to handle things properly. Bear with me on that

[674224c62f467voodoo47]

I'll move the few remaining mac related post to the mac topic as well, we want to keep things fairly organized. //done.

I still hope that Stephen comes around here soon ( he told me in private communication he would come here) to discuss the second source release.

sounds good - his previous account here is already expired, but we can reassign the existing posts to a new one as long as he registers again.

[674224c62f689Bert3]

...he told me in private communication he would come here...
Because as fun as it migt be, if this second source release is essentially around the corner - within the nexr weeks - then I don't see a point in trying to port the ported code back a second time.

would be nice to get an "official" statement - or least an info about what will be changed in the second source release

[674224c62f820dertseha]

This just in from Discord, by Stephen Kick himself:

I'll try to get a timeline of when you can expect a release of the code we've been working on
...
I'm being told around 4 weeks until we're done

and they also created a dedicated Discord channel, just for code-talk.

[674224c62f962voodoo47]

yep, I see it. maybe we should ask about the NDS port's renderer?

[674224c62faafvim]

I'll sign up to code for the project  . I say we focus on phase 0 (documentation) now and go to phase 1 (build setup) when we get the source port from NDS.

[674224c62fc1fBert3]

and they also created a dedicated Discord channel, just for code-talk.

link?

could someone add the discord stuff to https://www.systemshock.org/index.php?topic=10009.0 "Deck13/Welcome hacker!"

[674224c62ffa1dertseha]

link?

This Discord channel is on the NightDive Studios server. Since this is not a public community server, I'm hesitant to endorse it currently. A corresponding note (and link) has been added to the Welcome post.
It'd be great if people drew others to this forum here, not vice versa