The big dedicated server crash identification topic: UPDATED

This is the place where you can find everything related to the dedicated server, control scripts and community tools.

Moderators: Pit Crew, TM-Patrol

Post Reply
User avatar
BLaHiTiS
lord of the roads
lord of the roads
Posts: 3331
Joined: 12 Aug 2005 15:46
Owned TM-games: tmu tmo tms tmn
Manialink(s): muxitis
Location: Hasselt, Limburg, Belgium
Contact:

The big dedicated server crash identification topic: UPDATED

Post by BLaHiTiS » 23 Apr 2008 08:56

xbx wrote:Here is the dedicated for Trackmania United Forever and Trackmania Nations Forever.

You can download it from
http://files2.trackmaniaforever.com/Tra ... -05-19.zip
or
http://files2.trackmaniaforever.com/Tra ... -05-19.tgz

(both contain the same files, and windows and linux executable.)

Changes:
  • AutoSaveReplays is no longer ignored when using SetServerOptions.
  • StartServerInternet() doesn't work like in TMU/TMN -> return an error instead of pretending everything is fine...
  • A few settings that influence the server ressource usage (SetMaxPlayers, SetMaxSpectators, EnableP2PUpload, EnableP2PDownload, AutoSaveReplays, AutoSaveValidationReplays) are now reserved to the SuperAdmin.
  • Avoid to waste server P2P and cache by downloading the skins, avatar and lightmaps that everybody already has. (the ones given with the game)
  • other little bug fixes (like the server taking hours to launch if there are many autosaved replays.)
from 2008-05-07:
  • crash when a tcp connection is broken under certain rare conditions.
  • server restart no longer wipes the guestlist.txt and blacklist.txt
  • ForceSpectatorTarget() now works with cameratype = 2 and no target
  • Cup Mode: the game no longers loops when there's a winner and everybody else is disconected or in spectator mode.
  • use fallback timer for FreeBSD
  • detailed playerinfos work for the server login
  • new functions SetBuddyNotification() GetBuddyNotification() to be able to disable the chat message. (either per player or for all the players on the server)
Well,

A certain amount of people have been plagued by crashes of the dedicated server on their linux OS since the release of tmu-f.

It's a crash which is triggered somehow. I'll share my experience and we can pit this against all the other experiences and formulate a decision.

Allthought I "think" it is triggered by some events, this topic is needed because the crash is very evasive and isn't a direct result of server operation by the user but might be caused by records script events. Let's hope that there's a singular action you can take to make the server crash. But we also need to face the posibility that the crash happens in several phases.

There might be 2 or more criteria needed to let the server crash.

So let's put us all in the right direction at least.

If you participate in this topic, please make sure you are using 2008-05-19. As some fixes were done to evade other crashes that are not relevant to this issue. If you are using an older version, you might disrupt this poll cause the crashes may not be related to this particular issue.

so only test with 2008-05-19 please, if you want to participate.

You might use these versions and have used your "old" configuration files which is missing some data. It can be significant for this test or it may not.. We'll have to see the result.

So the things you might consider posting feedback on is:

Crashes regularly or no crash ?
Windows or linux
Do you run with old config files (2008-04-15 etc)
Do you start the server with old matchsetting files ?
Is your server having 60+ users regularly ?

Tell a bit about your config, do you autosave replays ?
Do you use different than standard upload/download rates etc..

Do you or don't you run server scripts ? In case you do:
Do you run Xaseco
Do you run Onyx 2
Do you run Fast 3
Do you run Aseco 2
Do you run aseco 1.2 ?

It's a vast examination I know, and even i'm not sure this is the right direction, other angles may be discussed in this topic too ofcourse.

I'm no programming guru, nor do I have intimate knowledge of the inner workings of the server like xbx does. Still i'd like to also bring the discussion forward on incomplete array data (for callbacks or configuration servers) whether they might or might not cause the server to behave badly and crash. That's why I'm asking you if you're using older scripts that were designed for TMU.

And that's why I'm also calling out to the script coders to tell if they have followed the new methods standards, and well.. I'd also like to ask if users of "new" scripts still have troubles.

If it's really internel, network tripping or other internal stuff of the dedicated beyond our immediate control, then this topic may be quite useless and it will be just a case for Nadeo to find this. But at least we've tried.

So here we go, I hope we can have a little of that beta test spirit out in the public :)

It seems that lots of the crashes are already resolved by the latest server, but you never know what you may find so this topic still has it's value.
Image

  • MB:Asus P8 H61
  • CPU:Intel i5 2400
  • RAM:8 Gb
  • GFX:Asus GTX-580 (1500mb DDR5) - Forceware 75.33
  • DirectX: June 2011
  • SND:Realtek HD on mobo
  • OS:Windows 7 SP1- Home Premium
[/size]

Xymph
Pit Crew
Pit Crew
Posts: 5701
Joined: 19 Aug 2007 12:58
Owned TM-games: TMN, TMU, TMF, TM²
Contact:

Re: the big dedicated server crash identification topic

Post by Xymph » 23 Apr 2008 09:16

Good idea.
BLaHiTiS wrote:so only use : 2008-04-16, 2008-04-18 please, if you want to participate.
XASECO explicitly tests the server build and will only run on 2008-04-18 (or newer).
BLaHiTiS wrote:And that's why I'm also calling out to the script coders to tell if they have followed the new methods standards, and well.. I'd also like to ask if users of "new" scripts still have troubles.
What exactly do you mean by "new methods standards"?
Developer of XASECO for TMF/TMN ESWC & XASECO2 for TM²: see XAseco.org
Find your way around the Mania community from the TMN ESWC hub, TMF hub, TM² hub, and SM hub

User avatar
BLaHiTiS
lord of the roads
lord of the roads
Posts: 3331
Joined: 12 Aug 2005 15:46
Owned TM-games: tmu tmo tms tmn
Manialink(s): muxitis
Location: Hasselt, Limburg, Belgium
Contact:

Re: the big dedicated server crash identification topic

Post by BLaHiTiS » 23 Apr 2008 09:20

well, you know, in the beta we had this problem that when you would return incomplete methods (like giving the the server configuration method from tmu to tmu-f dedicated ), it would destroy the referree password, maybe there's stuff like that in other arrays you can give back also ?
Image

  • MB:Asus P8 H61
  • CPU:Intel i5 2400
  • RAM:8 Gb
  • GFX:Asus GTX-580 (1500mb DDR5) - Forceware 75.33
  • DirectX: June 2011
  • SND:Realtek HD on mobo
  • OS:Windows 7 SP1- Home Premium
[/size]

Xymph
Pit Crew
Pit Crew
Posts: 5701
Joined: 19 Aug 2007 12:58
Owned TM-games: TMN, TMU, TMF, TM²
Contact:

Re: the big dedicated server crash identification topic

Post by Xymph » 23 Apr 2008 09:42

BLaHiTiS wrote:well, you know, in the beta we had this problem that when you would return incomplete methods (like giving the the server configuration method from tmu to tmu-f dedicated ), it would destroy the referree password, maybe there's stuff like that in other arrays you can give back also ?
Okay, I guess that's possible, but not with the SetServerOptions method in XASECO, as I don't use that one. :)
I'll do a check for other changed methods though.
Developer of XASECO for TMF/TMN ESWC & XASECO2 for TM²: see XAseco.org
Find your way around the Mania community from the TMN ESWC hub, TMF hub, TM² hub, and SM hub

User avatar
Onice_
wheelbarrow operator
wheelbarrow operator
Posts: 17
Joined: 19 Apr 2008 15:34
Owned TM-games: TMU, TMUF, TMN

Re: the big dedicated server crash identification topic

Post by Onice_ » 23 Apr 2008 09:46

Good idea !
May his knowledge database help us :p

Crashes regularly or no crash :

About twice a day

Windows or linux

Linux debian

Do you run with old config files (2008-04-15 etc)

I tried :
- version 2008-04-16 + only executable from version 2008-04-18
- full version 2008-04-18
same results, crashes

dedicated.cfg
<connection_uploadrate>8192</connection_uploadrate> <!-- KBps -->
<connection_donwloadrate>8192</connection_donwloadrate> <!-- KBps -->

<force_ip_address></force_ip_address>
<server_port>2354</server_port>
<server_p2p_port>3454</server_p2p_port>
<client_port>0</client_port>
<bind_ip_address></bind_ip_address>
<use_nat_upnp></use_nat_upnp>
<use_proxy>False</use_proxy>
<proxy_login></proxy_login>
<proxy_password></proxy_password>

<xmlrpc_port>5004</xmlrpc_port>
<xmlrpc_allowremote>True</xmlrpc_allowremote>
I dont know the utility of these :
<referee_password></referee_password>
<referee_validation_mode>0</referee_validation_mode> <!-- value is 0 (only validate top3 players), 1 (validate all players) -->

<use_changing_validation_seed>False</use_changing_validation_seed>
Do you start the server with old matchsetting files ?

I tried :
- start with original nadeo files
- start with TMWeb generated files
no difference for crashes

Do you use different than standard upload/download rates ?

First I tried standard rates.
Now :
Upload : 8162kB/s (bandwidth 12500kB/s)
Download : 8192kB/s (bandwidth 12500kB/s)


Is your server having 60+ users regularly ?

between 40 and 60 players, maximum players number is 60.

Do you or don't you run server scripts ?

I tried aseco 1.2 and aseco 2. Same results (maybe faster crashes with aseco 2 but not sure)

Now, I am trying Xaseco 0.96. I started it at 1h00 AM this night. About 30 players all the night, serveur full (60 players) since 9h00 AM. No crash for the moment, time will tell ...
EDIT : just crashed at 1h00 PM, server full, 12 hours uptime :/

Other remote tools ?

- I am using remoteCP3 to add maps and manage my server. I will update it to remoteCP4. Maybe it can helps

- Our website displays servers infos (current map, playlist and list of players), in read mode (only "get" methods)

Other things?

We have an old dedicated server TMN running on the same hardware server, differents ports, different aseco (Xymph version 0.93), different mysql database ...
Last edited by Onice_ on 23 Apr 2008 11:28, edited 3 times in total.
Member/Admin of the FullSpeedAddicts freaks !

Moriddin
road tourist
road tourist
Posts: 77
Joined: 21 Mar 2008 13:29
Owned TM-games: TMN, TMU, TMUF

Re: the big dedicated server crash identification topic

Post by Moriddin » 23 Apr 2008 09:58

Just something I thought was strange in the GBXRemote.inc.php when I wrote a pascal version of the transport protocol.

Code: Select all

$request = pack('VVa*', strlen($xml), $this->reqhandle, $xml);
First : I never find any reference to UTF-8 encoding of the $xml variable (but this can be my limited nowledge of php).

Second: If the $xml string is in UTF-8 then strlen($xml) gives the number or characters send, not the number of bytes (or is the length of the string expected).

Maybe just a long shot since it never went wrong in the old servers with the same code but if the crash its related to this then there could be problems with a lot of players with unicode characters in their names.

Pesky
smooth traffic navigator
smooth traffic navigator
Posts: 262
Joined: 19 Mar 2008 16:25
Owned TM-games: TMN, TMUF
Manialink(s): rrmania
Contact:

Re: the big dedicated server crash identification topic

Post by Pesky » 23 Apr 2008 11:11

Crashes regularly or no crash :

No Crash yet since Forever Release, using latest Dedicated Binary

Windows or linux

Linux debian

Do you run with old config files (2008-04-15 etc)

Using config File shipped with latest Binary
Using Standard Game Ports except 5033 for xmlrpc
P2P up and download is disabled


Do you start the server with old matchsetting files ?

Using Matchsettings-File shipped with latest Dedicated

Do you use different than standard upload/download rates ?

Standard rates.

Is your server having 60+ users regularly ?

Max is 40 Players. Quite often Full

Do you or don't you run server scripts ?

Running Mistrals Script in V7 with some minor selfcoded modifications

Other remote tools ?

Using RemoteCP3 and RCP4

Other things?

Stats on our Webpage

User avatar
Onice_
wheelbarrow operator
wheelbarrow operator
Posts: 17
Joined: 19 Apr 2008 15:34
Owned TM-games: TMU, TMUF, TMN

Re: the big dedicated server crash identification topic

Post by Onice_ » 23 Apr 2008 17:22

I have just noticed that when you extract the zip file "TrackmaniaServer_2008-04-18.zip", the following files "DedicatedTrackMania.TrackMania.gbx" and "NationsList.xml" (in GameData directory) are read-only ( -r--r--r-- ).

May these files can be modified by a running server?
If yes, maybe we have a start of explaination ... I hope.

NB : I have noticed that the file " DedicatedTrackMania.TrackMania.gbx" of a TMN/TMS/TMO "TmDedicatedServer_2006-05-30.zip" is with write rights ( -rw-r--r-- ) by default.
Member/Admin of the FullSpeedAddicts freaks !

User avatar
Dawa
wheelbarrow operator
wheelbarrow operator
Posts: 15
Joined: 21 Apr 2008 18:43
Owned TM-games: TMU, TMN
Contact:

Re: the big dedicated server crash identification topic

Post by Dawa » 23 Apr 2008 17:30

FYI, we moved the TMN server on another dedicated server, to let only the TMNF server on the dedicated, and same result, crash after 5/6 hours...
Member / Admin of the FullSpeedAddicts freaks !

User avatar
Slig
Pit Crew
Pit Crew
Posts: 2124
Joined: 05 Sep 2005 17:51
Owned TM-games: ALL
Location: TraxicoLand (Fr)
Contact:

Re: the big dedicated server crash identification topic

Post by Slig » 23 Apr 2008 20:17

Xymph wrote:
BLaHiTiS wrote:well, you know, in the beta we had this problem that when you would return incomplete methods (like giving the the server configuration method from tmu to tmu-f dedicated ), it would destroy the referree password, maybe there's stuff like that in other arrays you can give back also ?
Okay, I guess that's possible, but not with the SetServerOptions method in XASECO, as I don't use that one. :)
I'll do a check for other changed methods though.
Fast don't use SetServerOptions or SetGameInfos at all, but only individual SetXXX methods, and always use the Getxxx with Forever option for methods having it.

User avatar
BLaHiTiS
lord of the roads
lord of the roads
Posts: 3331
Joined: 12 Aug 2005 15:46
Owned TM-games: tmu tmo tms tmn
Manialink(s): muxitis
Location: Hasselt, Limburg, Belgium
Contact:

Re: the big dedicated server crash identification topic

Post by BLaHiTiS » 23 Apr 2008 22:34

Crashes regularly or no crash ?
No
Windows or linux
linux debian etch x64
Do you run with old config files (2008-04-15 etc)
nope, added the new options from the latest dedicated
Do you start the server with old matchsetting files ?
no (and those are always auto actualized when saving matchsetting in more modern versions)
Is your server having 60+ users regularly ?
yes
Tell a bit about your config, do you autosave replays ?
autosave replay=on,
Do you use different than standard upload/download rates
etc..
60000/60000
Do you or don't you run server scripts ? In case you do:
None

with this configuration, the server is already running 3 days and 6 hours straight without crashing.

I used remotecp4 to change maxplayers twice, I always am inserting new tracks with it from tmx, banned users, changed p2p settings (disabled upload, download was alreday disabled in my config)...

I saved matchsettings several times, all without aseco running.

When I used to run aseco 1.2.4 the server used to crash after 5 or 6 hours, also the latest version.

I havn't tried aseco 2 yet. I will do that tomorrow. (with running process)

So to recapitulate:

nations server now running fine for 3 days without aseco, but used to crash when many players and aseco were on.
United server runs well but nobody really visits the server. It has aseco 2.0 running there now.

A side note: what I also am wondering about if when you would use the same blacklist.txt and guestlist.txt, the server would crash also, . This has been resolved in my new configuration, my tm united server saves to different files, my nations server has no settings there so it saves to default ?

I hope when many people keep posting their experiences we can home in on the problem, and possibly do a next phase by stripping aseco down plugin by plugin (if that's possible), to know which plugin may cause it.

On the other sites i'm already doing tests with the help of slig's programming to tackle the server with constant callbacks, it hasn't fainted yet, and the requests get higher and higher.

in another phase we can do this with constand methods as well maybe, but i'm running ahead. Let's first hear everyon's story.
Image

  • MB:Asus P8 H61
  • CPU:Intel i5 2400
  • RAM:8 Gb
  • GFX:Asus GTX-580 (1500mb DDR5) - Forceware 75.33
  • DirectX: June 2011
  • SND:Realtek HD on mobo
  • OS:Windows 7 SP1- Home Premium
[/size]

Kjell T. Ring
sunday driver
sunday driver
Posts: 71
Joined: 03 Aug 2007 09:31
Owned TM-games: TMN, TMS
Location: Norway

Re: the big dedicated server crash identification topic

Post by Kjell T. Ring » 23 Apr 2008 23:16

Windows or linux

Win XP SP2 Norwegian
2008-04-18 TM Server
Not a sign of a crash. Running very stable with all released versions since the first (2008-04-15)

Do you run with old config files (2008-04-18 etc)

Using config File shipped with latest Binary
Running two servers from same .exe and subdirs using ports 2801 and 2802
P2P up and download is enabled 2811 and 2812.


Do you start the server with old matchsetting files ?

Using Matchsettings-File shipped with latest Dedicated

Do you use different than standard upload/download rates ?

Uploadrate: 4096
Downloadrate: 8192

Is your server having 60+ users regularly ?

Max is currently 50 Players. But our clan are guests, so the server often has 65++ players.

Do you or don't you run server scripts ?

XAseco 0.96 (0.95 also ran without problems)

Other remote tools ?

Using RCP4 (4.0.0.9)

Other things?

TMN Aseco Stats v1.7
Using Windows Firewall with following ports both UDP and TCP opened: 2801, 2802, 2811, 2812, 80
Xampp Win32 v1.6.6a

Zahl
happy cruiser
happy cruiser
Posts: 114
Joined: 18 Jul 2006 22:37
Owned TM-games: TM, TMU
Location: Germany
Contact:

Re: the big dedicated server crash identification topic

Post by Zahl » 24 Apr 2008 06:38

Moriddin wrote:Just something I thought was strange in the GBXRemote.inc.php when I wrote a pascal version of the transport protocol.

Code: Select all

$request = pack('VVa*', strlen($xml), $this->reqhandle, $xml);
First : I never find any reference to UTF-8 encoding of the $xml variable (but this can be my limited nowledge of php).
That's the programmer's job, the game needs it like that, so you have to pass utf8 strings to the class.
Second: If the $xml string is in UTF-8 then strlen($xml) gives the number or characters send, not the number of bytes (or is the length of the string expected).
No, it returns the number of bytes.
Intel Core 2 Duo E4400 @ 2.7GHz; 4GB RAM Corsair; Gf8800GTS; WinXPSP3; Samsung 223BW

Moriddin
road tourist
road tourist
Posts: 77
Joined: 21 Mar 2008 13:29
Owned TM-games: TMN, TMU, TMUF

Re: the big dedicated server crash identification topic

Post by Moriddin » 24 Apr 2008 07:09

Zahl wrote:That's the programmer's job, the game needs it like that, so you have to pass utf8 strings to the class.
True, but what I said was that in the construction of the $xml variable (which should be in UTF-8 format I gues since this is specified in the xml header tag) there is never any conversion to UTF-8.
Zahl wrote:No, it returns the number of bytes.
No it returns the number of characters in the string (nr of bytes is not interesting for string manipulation) Noted here : http://nl3.php.net/manual/en/function.strlen.php#72274.

But the only reason for a crash with this is when nadeo changed the xml parser or transport handler since the TMU dedicated (which should be easely checked).

User avatar
Onice_
wheelbarrow operator
wheelbarrow operator
Posts: 17
Joined: 19 Apr 2008 15:34
Owned TM-games: TMU, TMUF, TMN

Re: the big dedicated server crash identification topic

Post by Onice_ » 24 Apr 2008 12:17

This night, I configured a new server from scratch :
- new install of my OS (debian etch 4.0r2),
- no service installed (like php or apache)
- new dedicated trackmania server
- and nothing else

The server ran 12 hours and crashed ... I have no more idea :/

May it is possible that I have too much players on my server ? (100 players and 15% cpu usage ..., but I tried with 40, 50, 60 ... same result)
Member/Admin of the FullSpeedAddicts freaks !

Post Reply