It will help you to improve your skills or how to understand how to start this journey!
Welcome back for the second episode of this hate story between me and The Sabouter. Yes it isn’t a love story as it is not ended yet but in a week there was a lot of new things on the plate.
If you missed the previous episode that explain the project and the first step, check here.
The last week the lasts words was about tests the other Lua minors versions and do some analysis, but this week there was also a leak of 2008 and other updates.
I tried on xbox360 changing the localization files for the Italian language from the pc version on Xbox 360, but it was crashing. Using just the file for the localized strings was working, but it was using the wrong strings, probably the internal references was different. Anyway this was confirming that between the various release a lot of stuff was in common.
The leak
As I shared this article on reddit I was reached by 2 different people for 2 different things.
The first case about that point is that in 2009 2 weeks before the release of the game there was a game leak of the Xbox 360 version of the game itself. This leak is possible to download it from Alpha Archive, a collection of leaks and I shared it with the Discord Saboteur modding community.
The firsts effect was that no one in the community known about this (also after all these years) and created a big momentum, new channels and a lot of discussions about the data structures. I can say that the leak contains all the game assets not packaged, a lot of unused stuff and tests stuff, but it is very different from the final release.
All the xex binaries for Xbox360 doesn’t run on Xbox360 (I tested it on mine and all of them crashes). That leaks contain a lot of different builds with the debug symbols, so I was able to confirm Lua 5.1 and also that the savegame implementation was created as a custom module inside Lua in C++.
As you can see there are various binaries and debug symbols but they are all xbox 360 stuff that don’t run. Also Wildstar was the codename of the game engine mentioned in a lot of stuff in the leak and also in the binaries itself.
We cannot thrust the Lua bytecode, also if it is a header that specify the lua version itself. Searching on internet I saw that usually this value is tweaked to create problems. But there wasn’t so many reference to 5.1 and also that the bytecode generated was very similar with that version.
The rest of the community enjoyed looking at the various different textures and assets unused, but I wasn’t interested on this stuff. A user published on Youtube the various video included in the leak itself.
One of the member that is also the founder of the community, with the AlphaArchive community was able to get another dump. It seems that one of the members of that community got an Xbox devkit from the company that created the game, but the files were deleted, so they just recovered the filenames and no useful content (garbage).
Around there is also a E3 build from 10 days before of the release, but it is not yet possible to get it. The community that have those kinds of builds are using anonymous nicknames, and they are very jealous of what they have.
Ghidra
At the same time I was decompiling the saboteur pc version and Xbox 360 version and comparing them. As the file size is the very similar our guess is that the game binary includes how to parser some things, but the rest is in the various assets files. Like a package for sounds, one for video, one for animation, few for textures and one for game code.
I did some tests and the other user from Reddit reached me to share his Ghidra plugin that let to decompile the Xex binaries of Xbox360.
The issues were that it was just for Windows, so we discussed, I did a pr to compile a plugin internal tool also for Linux but the next morning a new release was available and this time was in pure Java.
Let me explain better, Xex, that sound similar to Exe, are the binaries for Xbox360 that are crypted/compressed with a modded LZ compression algorithm. Also, the headers of those files have a mention that aren’t Win32 binaries. This is important because the leak include exe, xex and other files but at the end they was all Xbox 360 stuff. The plugin before my request was using a binary to do that conversion, but the author found a Java implementation so it was possible to remove that stuff and get a plugin that was now cross platform (hurrah for linux!).
With this plugin I was able to inspect the Xex binaries that gave nothing useful to me apart the fact that the game include a lot of strange things, like if that the binaries wasn’t cleaned before publishing.
This screen is from the pc version and there are a lot of references of that, seems that was the path of the build machine but doesn’t make so much sense to have this information in the binary itself, in a variable. Anyway this stuff there isn’t on Xbox 360. It was helpful to give some hints how the Wildstar engine was pure C++ with the lua integration.
Not all the game code was written in Lua but inside the binary expose those things to Lua itself.
There was those reference in the binaries that are the method used in the Lua code (that we are able to get as in the first artcile of this series) and the other in the screenshot to manipulate the savegame.
Also the same user released a tool to connect Ghidra to a running Xbox 360 with the game to inspect what is happening, but I didn’t still use it.
Lua Drama
So after all this information it was clear that the key is to have the same bytecode so we are able to replicate the lua building system. In this way we can patch the game for xbox with different code.
I don’t know Lua, I just saw a bit in NeoVim but this project required to learn a lot about it.
So I discovered:
- Lua is used in a lot of games in all the kind of devices because the engine is just (few) Ansi C files with no dependencies, in this way it is easy to mod but also to integrate in other things
- It is used a lot in the embedded devices in the bytecode form to improve the performance of 10x
- Every Lua major version change the syntax with the bytecode, so you need to use the same version of your target also if has over 10 years
- This version schema let the language to be fixed in the time with no changes in the syntax, in this way is used in critical context and in others areas like games
- Lua 5.1 last version was released in 2009 and a lot of tools for that version are for this period, also if a lot of games (also recent) still uses old versions (the last one is 5.4)
So my first hint was, if a lot of developers are working with Lua on router and firmwares they wrote or did some experiments on reverse engineering!
So I found this article of 2020 with a tool that targeted Lua 5.1/5.2 and was able to inspect the bytecodes.
In this 2 screens you can see the difference between the bytecode of the same file from one file of the leak and seem file bytecode generate with my tool.
So I have chosen to compile all the 5.1.0/1/2/3 versions for windows and see if the bytecode is different as I closed the first episode of the series. Anyway the bytecode was the same for all of them for the same file, a lot of time on getting a Windows VM, build scripts, get GCC on Windows XP (I was trying to emulate the dev system there were using) for nothing.
It is clear that the difference it is in the bytecode structure itself but is not for the Lua interpreter version. Also I saw something that I forgot totally, the game was built for xbox but the bytecode is different on CPU bases.
Xbox 360 is based on PowerPC64 that is a big endian CPU instead x86-64 is little endian.
Later I found this website that enable to load a Lua bytecode and show some information about it without use an external tool.
It was time to learn a bit how Lua bytecode works…
The first was that luac (lua compiler) doesn’t support crosscompile, that maybe you can achieve it with a custom luacomp.c modded version, there is some code around, but it is very hacky and unclear how to do this changes. Anyway on Lua mailing list they just say that you need to change the Lua code to achieve cross compiling and it is not very suitable to me.
I discovered a web tool to compile Lua just for the version that you want from the browser but anyway this wasn’t the tool I was looking for. In the meantime I asked the Lua reddit community to see if I was able to get some hints. The only one I got was to use Qemu and get a ppc64 machine to emulate lua and generate it from here.
So I got the idea maybe I can convert the bytecode for another architecture without have that architecture!
Chunkspy is an old Lua tool, the last version is from 2006 and let you to convert a bytecode rewriting it with Lua itself.
With a web viewer I was able to compare the difference and it wasn’t just the endianess requiring a change, but also other header info.
Chunkspy is a script created to be modded so you can create your Lua header settings, I created a tiny diff that on install of my suite patch Chunkspy with the new PowerPC64 support for this tool.
I converted the bytecode and… it wasn’t the same but similar. Anyway it was very very similar compared to before with just some offset that are like the half of the value itself.
In this 2 screen you can see just the few differences (there are just another few) between the original and the final generated version (the game crashes also with that file). Later I patched the issue of D:\ that is lowercase but it was very simple.
As you can see the difference in the bytecode are just a different lines position for the bytecode viewer, that to me doesn’t make any sense. But a lot of things doesn’t make sense to me. Like Ananas on Pizza, but this is another story.
During my research I also discovered why it is used D:\ as drive for Xbox in Lua, it is the default drive also if in Xbox that one doesn’t exist, it is like a placeholder. The article was very interesting as it is of 2009, sadly there is a zip file that should include tools and some examples and a patched lua (!) but is not available anymore..
This thing opened the eyes to me, maybe on Xbox 360 there are already tools with Lua and I can use the interpreter they have to execute my Lua and generate a valid bytecode!
On modded Xbox 360 there is a custom dashboard that let’s to launch other games with a lot of features, it is isn’t open source (I am using it) but is scriptable with Lua!
There is also a console that extend this integration but… everything it is for Lua 5.2. Also I don’t know how it is possible to generate lua bytecode from lua itself anyway…
So I asked to me maybe there is a Virtual machine for Lua written in lua that target Lua 5.1, as they are just different syntax subset supported. I found Lua VM, not updated since years and let you to generate bytecode as big endian instead of converting, but just if the lua code is in asm and not generic lua code. So I cannot using it also because it is incomplete so is not thrustable but was letting to target the Lua version.
The other sad news that the official Lua website mentioned various other Lua implementation but the code is not avalaible on internet anymore for the ones wrote in Lua itself.
The last thing that I tried are to cross compile lua for PPC64 but I cannot run it without a VM (yet). To achieve that I also did 2 pull requests to the Free60 project wiki (there was some markdown issues) that in the past was able to install Linux on the Xbox 360.
Conclusion
At the second week of the project I am very tired, as I don’t like to have a VM to generate a stupid bytecode!
I don’t understand why there isn’t any cross compiler or converter that can do the same bytecode…
I don’t know if there will be the third part (or when) but I will leave here the next steps that is possible to do.
- Xenia emulator now supports better the game, also if crashes after a while. It can be useful to speed up the tests also if it is only for windows
- The modding community is working to reach the various original game developers to see if they have more files or devtools (it will be amazing)
- Execute the lua compiler for PPC64 on a PPC64 machine (also emulated) and see the bytecode generated (I need help for this as I never used Qemu)
- Connect the original Xbox 360 for remote debugging with Ghidra, to get the real error. It could be useless as if the bytecode is different there will be some not valid chunk or errors like that.
I want to say thank you to all the Discord The Saboteur modding community, for the help, hints and discussion. I prefered to not mention their names in the article but I will say here their nicknames: Happys, Gamelaster, Blumster, R3sus and of course Dan.
Also thanks to WarrantyVoider for his Ghidra’s tools and help.
You can find me on Telegram, Discord, Reddit, Twitter and so on just check with Mte90 nickname or on the blue sidebar on the right.
