Bug #1243

Server crash related to filesystem access

Added by Spyhawk 12 months ago. Updated 4 months ago.

Status:New% Done:


Target version:2.77
OS: Arch:


The test server tends to crash every few weeks or so. The logs have interesting content:

  • it seems the game log file can’t be opened (maybe some issue with log rotation)
  • the mapvote.txt suddenly can’t be opened
  • the database eventually can’t be accessed anymore (locked), resulting in a hard crash
  • There are some WA error, but I believe this is unreletad (WA got some bugfix in its dev branch)

See attachment, last bit of the ~80mb server log file.

output.log.txt Magnifier (751 KB) Spyhawk, 02.04.2019 18:23

etlded-fds.txt Magnifier - Ododo's file (1.73 KB) Spyhawk, 26.11.2019 12:25

Related issues

Related to ET: Legacy Development - Bug #987: Server crash Feedback 01.02.2017

Associated revisions

Revision c6d650ea
Added by Spyhawk 12 months ago

game: fixed mod logfile not always created, refs #1243


#1 Updated by Spyhawk 12 months ago

For reference, there is a copy of the three previous crashed logs on the server (internal access, 80+ mb each):
  • output_crashed_at_0820_on_270119.log
  • output_crashed_at_1112_on_010419.log
  • output_crashed_at_0554_on_060519.log

Additional info:
Restarted the test server.
`WARNING: Couldn’t open logfile: legacy_mod.log` is printed, but doesn’t seem to be critical here. (done)

#2 Updated by Spyhawk 12 months ago

About the mod log side-issue: it looks like the "append" mode fails to create a new file when it isn’t already present. For some reason, the "create" fallback just fails.
  • The issue doesn’t happen on windows, but only Linux (confirmed on the test server running CentOS 7, locally on Arch, and on by a third party on his server).
  • this isn’t a permission issue, since it is possible to create a new file (tested locally)

I couldn’t find much about that issue, apart from this article ("The descriptions for both the "a" and "a+" append modes say that the file will be created if it doesn’t exist. I’ve found, in practice, that this doesn’t work. If the file doesn’t exist when fopen is called with a "a" or "a+" open mode, the file will not be created.")

Suggestion for a dirty workaround: if the append mod fails, try creating the file with a normal "write" mode, close the file, and try to append it to it again. (done)

#3 Updated by Spyhawk 11 months ago

Server crashed again. Same symtoms. Log is 95 Mb this time (output_crashed_at_0453_on_190419.log).

A recent commit fixed some issue in the Lua stack (" Lua vm stack was uncontrollably growing after each hook
callback call"), see if that fix improves the situation on the next update.

  • 06/05/19: Ignore the following log: output_stopped_at_1820_on_060519.log (server was restarted for maintenance).
  • 05/06/19: new crash. Same symptoms. The Lua stack growth fix doesn’t seem to have improved the situation here.
  • 28/06/19: yada. Same symtoms.

#4 Updated by Spyhawk 11 months ago

  • Related to Bug #987: Server crash added

#5 Updated by Spyhawk 9 months ago

  • Target version changed from 2.77 to 2.78

#6 Updated by Spyhawk 5 months ago

  • Target version changed from 2.78 to 2.77

#7 Updated by Spyhawk 4 months ago

See Ododo’s comment on seemingly unrelated(?) issue: https://dev.etlegacy.com/issues/987#note-8

A file descriptor for etl.db seems to be created on each map load, without closing the previous. (see attachment)
So maybe the system has reached a limit as the errno 24 on mapvoteinfo.txt suggest.

Also available in: Atom PDF