Server crash related to filesystem access
The test server tends to crash every few weeks or so. The logs have interesting content:
- it seems the game log file can’t be opened (maybe some issue with log rotation)
- the mapvote.txt suddenly can’t be opened
- the database eventually can’t be accessed anymore (locked), resulting in a hard crash
- There are some WA error, but I believe this is unreletad (WA got some bugfix in its dev branch)
See attachment, last bit of the ~80mb server log file.
Restarted the test server.
`WARNING: Couldn’t open logfile: legacy_mod.log` is printed, but doesn’t seem to be critical here. (done)
- The issue doesn’t happen on windows, but only Linux (confirmed on the test server running CentOS 7, locally on Arch, and on by a third party on his server).
- this isn’t a permission issue, since it is possible to create a new file (tested locally)
I couldn’t find much about that issue, apart from this article ("The descriptions for both the "a" and "a+" append modes say that the file will be created if it doesn’t exist. I’ve found, in practice, that this doesn’t work. If the file doesn’t exist when fopen is called with a "a" or "a+" open mode, the file will not be created.")
Suggestion for a dirty workaround: if the append mod fails, try creating the file with a normal "write" mode, close the file, and try to append it to it again. (done)
Server crashed again. Same symtoms. Log is 95 Mb this time (output_crashed_at_0453_on_190419.log).
A recent commit fixed some issue in the Lua stack (" Lua vm stack was uncontrollably growing after each hook
callback call"), see if that fix improves the situation on the next update.
- 06/05/19: Ignore the following log: output_stopped_at_1820_on_060519.log (server was restarted for maintenance).
- 05/06/19: new crash. Same symptoms. The Lua stack growth fix doesn’t seem to have improved the situation here.
- 28/06/19: yada. Same symtoms.
- File etlded-fds.txt added
See Ododo’s comment on seemingly unrelated(?) issue: https://dev.etlegacy.com/issues/987#note-8
A file descriptor for etl.db seems to be created on each map load, without closing the previous. (see attachment)
So maybe the system has reached a limit as the errno 24 on mapvoteinfo.txt suggest.