WorkshopBackup Manual

From Clogopedia, the Natural Selection 2 wiki
Jump to: navigation, search

Editorial Note

Please keep all comments on the Talk page on the wiki, UWE forums post or on the NS2 Server Ops discord.
All information is current as written on 23-12-2017.
All information is verified as written on 23-12-2017.

Please keep a version number with the date included on big changes.

Version 23.12.2017.002

dd.mm.yyyy.change-amount

Credits & Thanks

  • Ghoul, for providing most of the info I did not directly copy/paste.
  • UWE (And whomever wrote the readme)
  • DCDarkling, me.


Intro and info

General Info

This manual will explain how to setup and use the WorkshopBackup for NS2 servers.

  • The WSB is technically a BETA.
  • Unknown problems can exist. The WSB is just better than nothing.
  • Due to the beta nature the default settings and setup will not grant the most benefit.
  • The WSB comes in 2 flavours which can be used by both Windows and Linux.
    • A 'Windows only' executable.
    • A Python script.
    • While harder to setup the Python script is easier to update and more stable.
    • It is HIGHLY advised to run the Python script.
    • The Python script uses version 3.4! Note this version, higher like 3.6 will NOT work.
  • It will use all, note all available bandwidth. It does not have a limiter.
  • Note depending on bandwidth, mod size etc, clients may need a few retries and reconnects.
  • You can get your most recent workshop backup server download here:


Goal of WSB

The goal of WSB is to provide a working backup for clients to download mods. Most often this need arises due to problems with the Steam Workshop.




Default

Default Setup

The WSB is shipped with both the client and the server. Lets focus on the server. The shipped version is in the following path:

\steamapps\common\Natural Selection 2 Dedicated Server\utils\WorkshopBackup

Running it from this path or a custom path produces the following situation. Note that the WSB is in a separate location compared to the location the server stores its mods.

WSB Default.png


Default Workings & related issues

Having the default setup causes the following scenario when the client can not contact the Steam Workshop, it will download files from the Workshop Backup Server.

WSB NormalWorkings.png


However this can cause a problem as shown in the next example. In this example the Steam Workshop is down for both the client and the Workshop Backup Server.

If the WBS did not download a requested mod in the past, it will not be able to retrieve the mod.

It can than not supply the mod to the client.

The client will than lack the required mod and connection to the gameserver will fail.

WSB WSDownWorkings.png


Thankfully the suggested or adviced setup will fix this key issue.




Suggested

Suggested Setup

A much better location to run the WSB server from is from the Servers OWN mod directory.

The default directory is located in:

%APPDATA%\Natural Selection 2\Workshop

If you use the following server variable, you will have set a custom location for your NS2 server mods:

-modstorage 	path

Running the WSB from the servers own mod directory will present you with the following situation:

WSB Suggested.png


'Suggested' Workings & fixing 'Default' related issues

Having the suggested setup causes the following scenario if the client and the Workshop Backup Server both can not contact the Steam Workshop, as shown in the example below.

It will still download files from the Workshop Backup Server.

WSB SuggestedWorkings.png
Note that if the WSB can reach steam workshop, the example from the default setup still applies as there is no problem.


Critical points & deeper workings of 'Suggested' Setup

You need, read this well, NEED to use the latest version of WSB from Ghouls github, as well as running the Python script and NOT the executable.

Remember that if Steam Workshop is down you have no source for the Workshop Backup Server to download mods from.

The solution to this problem is using the server itself to be a source for mod downloads.

If we run the WSB out of the same directory as the server, it will zip any mod it comes across. Steam Workshop & WSB supply archive zip files of mods. So zipping existing mods, if using the same kind of zip, produces the same modfile for download.

Yes this does mean you can technically get away with not running WSB from the servers mod dir, if you manually zip mods and supply it to the WSB install directory.
However by doing this you need to be sure that the method with how you make the zip file produces the same result unzipped as to not break consistency checks.
Due to the beta nature of the WSB, currently there is no known (public?) list of methods to do this. Caution adviced.
A reminder that using the server as source for mods obviously means that the WSB can only supply mods that the source server has in its directory!
More simple put; If the source server does not have the mod, the WSB does not have the mod.
This is of course excluding any mod the WSB already has stored due to other methods like Steam Workshop downloads or server operator actions.




Testing the WSB

Testing WSB outside NS2

If you have a zipped mod inside the WSB directory, you can use a weblink to download the mods zipfile through the browser.

If testing locally on the same server
http://localhost:27020/m5f3f7c0_1348957670.zip
http://Servername, usually localhost will do:port/zipfile of mod to test
If testing remotely
http://http://server.example.com:27020/m5f3f7c0_1348957670.zip
http://Server URL or IP:port/zipfile of mod to test


Testing WSB on the NS2 Client

To test on the NS2 client enable verbose 1 in the console before connecting to your server.

The best testing is done from a external location.

You should see the following info in your console and client log.

Enabling verbose

verbose level: 1

Error if a mod has no valid Steam Workshop Download location

[  8.278] MainThread : Error: Mod '"Shine Administration"(706d242)' doesn't contain a valid download url

Unmount messages (In nonverbose Mount messages show, unmount does not)

[ 17.235] MainThread : Unmount C:/Users/WindowsUserName/AppData/Roaming/Natural Selection 2/Workshop/m2b438822_1468700872/

After this the really interesting stuff starts as shown below.

[ 23.427] Worker 15 : Mod "Modname"(Mod ID): Client Mod is same version as on server
[ 23.427] Worker 15 : Mod "NS2+"(334982d2): found correct local version '1512585193'
[ 23.427] Worker 15 : Starting download of '"[Mod File]"(Mod ID)' from 'Remote File & File location'
[ 23.427] Worker 15 : Starting download of '"[m33aee7f1_1511560579]"(33aee7f1)' from 'htp://45.35.192.50:27020/m33aee7f1_1511560579.zip' (Note I broke this link on purpose in this example so its coloured, in console and log it shows a real link)

[ 44.317] MainThread : Mod "[Mod File]"(Mod ID): Notification download ended
[ 44.317] MainThread : Mod "[m33aee7f1_1511560579]"(33aee7f1): download op ended
More clear notification download ended
[ 45.483] MainThread : Download of '"DMD-Unstuck"(33aee7f1)' complete

Use this information to troubleshoot your WSB.

  • Does it connect to Steam or the WSB?
  • Does it try to even download files or has it a local cache?
  • Does the server have the file. (If the client does not, and there is no download, it could not be on the server.




Additional information

Mod info

Lets take the mod Shine Epsilon Beta as a example.

m1bbfa39e_1488119894

The format is Unique Mod ID_Unix Time

This means that any mod starting with m1bbfa39e is Shine Epsilon Beta.

The Unix Time part counts upwards.

This makes it so that you can have various zip files and folders of the same mod, but of different versions.

One of the reasons why its not as relevant which version the server runs compared to the client if Steam Workshop goes haywire is due to versions. It allows the WSB to supply the version the server uses, to the client.


Installation of the Python version of WSB

Acquiring the Python version

As stated, WSB works best with Ghouls most recent Python script version.

First acquire the 3.4 version of Python for your OS:

https://www.python.org/downloads/

Note this may contain only source code. You may need to look inthe Windows download section for prebuild installers.

I am blindly assuming the Linux Server Ops will just compile the latest 3.x.

https://www.python.org/downloads/windows/
Latest I found is 3.4.4

Running the Python Script

Note this is Windows only as I am assuming Linux Ops know how to do this.
Also my Linux knowledge is to limited to fill in.
Linux Ops, feel free to add any information.

The most simple is linking .py to start with the Python Launcher. After that just start the script.

However doing it like this may not be wishful if you wish to autostart it or use it with something like brainless panel.

You can do this by starting python with commands. A example where Python is installed on P: and the dedicated server on G:

python myscript.py
P:\Python34\python.exe "G:\Steam\steamapps\common\Natural Selection 2 Dedicated Server\Mods\workshopbackup.py"

Note that if your paths contain spaces, that you need to use " ".


Additional information & Settings like port

File info

After running the WSB once, a few files will be created.

  • workshopbackup.json - containing the settings.
  • log.txt - containing basic logging info.
  • log-err.txt - containing basic error info.

A additional start will rename some files and create some new ones.

  • log.txt is renamed to log.old.txt.
  • A new log.txt is created.
  • log-err.txt is renamed to log-err.old.txt
  • a new log-err.txt is created.
Note: in general usage the log files can get quite big.
You may want to consider disabling logs for running normally, after a day of testing.

Setting up the actual NS2 server for WSB

If the mod_backup_servers variable is not present in ServerConfig.json, start and stop the server, it should generate the default variables.

In ServerConfig.json, set mod_backup_servers to the list of backup servers.

"mod_backup_servers":[ "Serverlink or ip here", "second serverlink or ip here" ],
"mod_backup_servers":[ "Serverlink or ip here" ],
"mod_backup_servers":[ "http://example.com:27020", "http://example2.com:27020" ],

If you wish to make clients ignore Steam Workshop altogether set the following value to true in your ServerConfig.json.

  "mod_backup_before_steam":true,

WSB Settings

| Name                                    | Description                                                                     | Default                |
| ALLOWED_APP_ID                          | List of allowed app ids to store mods from. Empty = allow all                   | [4920]                 |
| ALLOWED_MOD_IDS                         | List all allowed mod ids to store. Empty = allow all                            | []                     |
| INTERFACE                               | Interface to listen on. 0.0.0.0 = all                                           | 0.0.0.0                |
| PORT                                    | Port the server will listen to                                                  | 27020                  |
| MAX_OUTSTANDING_STEAM_DOWNLOAD_REQUESTS | Maximum amount of downloads from steam the server will process at the same time | 4                      |
| VERSION_OVERLAP_WINDOW                  | How long we preserve a outdated version of a mod in secs                        | 604800  (1 week)       |
| LOG                                     | If True trace logging will be enabled                                           | true (change to false) |

Maintenance

The server should require zero maintenance, apart from possibly restarting the server every 6 month or so just to truncate the log files.


Mods are downloaded as clients submit requests for them - no need to manually update anything.


Old mods are cleared out about a week after the latest version comes out.


The server can be restarted at will - it will serve any existing file in the directory (that matches the m%x_%d pattern), so restarting does not mean it will have to re-download files from steam.


Stopping the server and removing all files from the directory will reset the server - possibly useful if there are too many old mods that noone uses anymore.


The log-err.txt contains both errors and a record of all download requests.


Expected resource usage

The server should be very resource-conservative. As it will only serve server-side mods, and clients will only connect to the server if they don't have the mod already, the only time it will see significant use is


  • first time players at a server missing one or more server-side mods (aka steam sale time)
  • when a server-side mod is updated.


Note that while most mods weighs in at the few-kb size, maps range in size from 5 to 50Mb.


Additional information about workings WSB

The server acts as a very restricted web server. It ONLY services requests for files whose pattern is "/m{mod-id-in-hex}_{timestamp/version-in-decimal}.zip".


If a file matching that pattern is present, it is served via the usual http-protocol.


If the request does not match the pattern, it is 404'ed.


If the request DOES match the pattern, but there is no such file, a 202 (request accepted, come back later) is returned and a request is sent to the steam-downloader part of the server.


The steam downloader will (in the background) try to download information about the given mod_id. Once steam answers, the response is analysed and stored away (persisted in the content.json file), and if the mod should be served, downloaded and stored in the current directory.


To avoid spamming steam too much, the server keeps track of invalid requests so multiple requests to outdated, unserved or otherwise invalid mods are denied efficiently.


This means that the set of mods being backed up will automatically adjust to the set of mods actually being used by servers backed up, without any need for manual updating.