executor: linux: fix syz_open_dev virtual file names range

We noticed that syzkaller left some files with fairly unusual file names
under /dev. Eg:

----------   1 root    root             0 May 30 14:42 vcs-
----------   1 root    root             0 May 30 14:48 vcs.
----------   1 root    root     136317631 May 30 14:42 vcs'
----------   1 root    root             0 May 30 14:48 vcs(
----------   1 root    root             0 May 30 14:43 vcs)
----------   1 root    root             0 May 30 14:43 vcs*
----------   1 root    root     136317633 May 30 14:46 vcs+

Funnily enough the characters after "vcs" are always within the '0'-10
to '0' ASCII range. We noticed that the syz_open_dev logic uses a modulo
10 on a signed number (the volatile long a1 argument) and in C the
modulo of a negative number stays negative, so the result of this
operation is in the '0'-10 to '0'+10 range. This is in turn casted to a
char which is also signed and doesn't fix the glitch.

By casting a1 to an unsigned long first, this keeps the result of the
modulo operation signed and therefore the virtual file name suffix a
number.

0 files changed, 0 insertions, 0 deletions