Remove binary diffing library that was moved to arcadeutils, move patches to their own directory.

This commit is contained in:
Jennifer Taylor 2021-06-27 17:08:15 +00:00
parent 9d6f753cb7
commit 61f3a570a7
16 changed files with 65 additions and 478 deletions

View File

@ -29,7 +29,7 @@ Images can be ripped and burned again using your favorite image reading/writing
## Utilities
Inside the `utils` directory you will find Python3 code that performs a variety of actions. The target version of Python3 I used was 3.6, but any version newer than this will work as well. The code is organized in a way that will hopefully promote reuse in other areas where it may be useful. On unix-like systems where Python3.6 or greater is installed, you can run these directly. On Windows, run them from the command line that has Python3.6 on the path by prefixing "python3" to the command.
Inside the `utils` directory you will find Python3 code that performs a variety of actions. The target version of Python3 I used was 3.6, but any version newer than this will work as well. The code is organized in a way that will hopefully promote reuse in other areas where it may be useful. On unix-like systems where Python3.6 or greater is installed, you can run these directly. On Windows, run them from the command line that has Python3.6 on the path by prefixing "python3" to the command. Do note that these utilities require the <https://github.com/DragonMinded/arcadeutils> repository be installed before running. The easiest way to do that is `python3 -m pip install -r requirements.txt --upgrade` at the root of this repository.
* `exe_utils` - Utilities for working with Firebeat EXE files. Run with `--help` to see full options. Note that PPP binaries are more complex, so when working with them be sure to use the `--ppp` flag.
* `exe_utils unpack` - Can take a `HIKARU.EXE` or `FIREBEAT.EXE` and unpack it to its raw PPC form, as described in the executable format above. This is suitable for decompiling or applying hex edits to change behavior or text.
@ -37,10 +37,6 @@ Inside the `utils` directory you will find Python3 code that performs a variety
* `exe_utils diff` - Can take two Firebeat EXE files and output the diff of their PPC code as a list of patch offsets. Use this to generate patch lists like you see below.
* `exe_utils patch` - Can take a Firebeat EXE file and a list of patch offsets and apply the patches to the EXE file. Use these to apply patch offsets found below to a Firebeat EXE.
* `bin_utils` - Utilities for working with raw PPC binaries that have been extracted. Run with `--help` to see full options.
* `bin_utils diff` - Can take two binaries and output the diff between them as a list of patch offsets.
* `bin_utils patch` - Can take a binary and a list of patch offsets and apply the patches to the binary file.
* `dallas_crc` - Simple utility that can replicate an iButton CRC for the laser-etched ROM ID written on the iButton itself.
* `dongle_dump_utils` - Utilities for crafting dongle dumper executables that run on a Firebeat.
@ -48,95 +44,6 @@ Inside the `utils` directory you will find Python3 code that performs a variety
* `dongle_dump_utils password` - Update an existing dumper executable to contain the passwords to dump a particular game.
* `dongle_dump_utils validate` - Perform a simple validation over a reconstructed dongle dump to check it for sanity.
## Patch Offsets
## Patches
The following are patch offsets that you can apply to a raw PowerPC image that has been extracted. The number on the left of the colon is the hex offset where you should make the change, and the numbers on the right of the colon are the before and after values at that location. To use any of these, obtain an image of the game, copy the Firebeat EXE out of the image, decompress it using the `exe_utils unpack` command, apply the edits using your favorite hex editor, recompress the image using `exe_utils pack` and then replace the Firebeat EXE in the image you obtained the original from. Alternatively, you can use `exe_utils patch` to patch a Firebeat EXE directly, or `bin_utils patch` to patch an unpacked PPC image. Patch offsets can be generated by diffing two Firebeat EXE files with `exe_utils diff` or by diffing two PPC binaries with `bin_utils diff`.
For convenience, many of the patches that appeared here have been moved into the `patches/` directory so they can be applied directly by `exe_utils` or `bin_utils` to executables after cloning this repository. No need to copy-paste or use a hex editor!
### Keyboard Mania
#### Skip Dongle Check
* 2C530: 7F C3 F3 78 -> 38 60 00 00
### Keyboard Mania 2ndMIX
#### Skip Dongle Check
* 51D53: 01 -> 00
* 51DC3: 02 -> 00
* 51E17: 03 -> 00
* 51E4B: 04 -> 00
#### Skip E940 Error
If you have used a donor board from a Pop'n Music or Beatmania III to do a mainboard repair for Keyboard Mania, sometimes it can throw an E940 error after passing all hardware checks. This seems to be an error verifying that the software is running on the right type of Firebeat. This skips this check and allows you to transplant parts between Firebeat boards to make a Keyboard Mania main board.
* 5A44: 41 82 -> 48 00
### Keyboard Mania 3rdMIX
#### Skip E940 Error
If you have used a donor board from a Pop'n Music or Beatmania III to do a mainboard repair for Keyboard Mania, sometimes it can throw an E940 error after passing all hardware checks. This seems to be an error verifying that the software is running on the right type of Firebeat. This skips this check and allows you to transplant parts between Firebeat boards to make a Keyboard Mania main board.
* 4F40: 41 82 -> 48 00
### Pop'n Music 4
#### Skip Dongle Check
* B8EC: 48 00 E8 E1 -> 38 60 00 00
* B914: 48 00 E7 4D -> 38 60 00 00
* B924: 48 00 E1 C9 -> 38 60 00 00
### Pop'n Music 5
#### Skip Dongle Check
* CCFC: 48 01 43 F9 -> 38 60 00 00
* EFD8: 48 01 21 1D -> 38 60 00 00
### Pop'n Music 6
#### Skip Dongle Check
* 509B6: FF FF -> 00 00
* 56498: 48 03 4A 4D -> 38 60 00 00
* 56520: 48 03 42 E5 -> 38 60 00 00
* 78A84: 48 01 24 61 -> 38 60 00 00
* 78ABC: 48 01 1D 49 -> 38 60 00 00
* 78BD8: 41 82 -> 48 00
* 8AF4A: FF FF -> 00 00
### Pop'n Music 7
#### Skip Dongle Check
* 5BEB0: 48 04 20 2D -> 38 60 00 00
* 5BF54: 48 04 3B 21 -> 38 60 00 00
* 858E0: 48 01 85 FD -> 38 60 00 00
* 859FC: 41 82 -> 48 00
* 8E622: FF FF -> 00 00
### Pop'n Music Mickey Tunes
#### Skip Dongle Check
* B924: 48 01 26 01 -> 38 60 00 00
* CD7C: 3D 60 80 10 -> 39 60 00 00
* CD80: 81 6B 02 94 -> 39 60 00 00
* DB10: 48 01 04 15 -> 38 60 00 00
* 1D0FE: FF FF -> 00 00
* 1E66A: FF Ff -> 00 00
### Pop'n Music Mickey Tunes Update Disk
#### Skip Dongle Check
* 30524: 48 01 D0 09 -> 38 60 00 00
* 31E0C: 3D 60 80 15 81 6B 6A 4C -> 39 60 00 00 39 60 00 00
* 32BB0: 48 01 A9 7D -> 38 60 00 00
* 43A76: FF FF -> 00 00
* 4DC72: FF FF -> 00 00
The patches that appear in the `patches/` directory are provided for others wishing to fix or change software on their Firebeat-based cabinets. They follow the patch format laid out in bindiff from <https://github.com/DragonMinded/arcadeutils>. The only difference is that the file is first decompressed before file size comparisons and byte modifications are made, and finally the file is recompressed again.

View File

@ -0,0 +1,2 @@
# Skip Dongle Check
2C530: 7F C3 F3 78 -> 38 60 00 00

View File

@ -0,0 +1,5 @@
# Skip Dongle Check
51D53: 01 -> 00
51DC3: 02 -> 00
51E17: 03 -> 00
51E4B: 04 -> 00

View File

@ -0,0 +1,7 @@
# If you have used a donor board from a Pop'n Music or Beatmania III to do a
# mainboard repair for Keyboard Mania, sometimes it can throw an E940 error
# after passing all hardware checks. This seems to be an error verifying that
# the software is running on the right type of Firebeat. This skips this check
# and allows you to transplant parts between Firebeat boards to make a Keyboard
# Mania main board.
5A44: 41 82 -> 48 00

View File

@ -0,0 +1,7 @@
# If you have used a donor board from a Pop'n Music or Beatmania III to do a
# mainboard repair for Keyboard Mania, sometimes it can throw an E940 error
# after passing all hardware checks. This seems to be an error verifying that
# the software is running on the right type of Firebeat. This skips this check
# and allows you to transplant parts between Firebeat boards to make a Keyboard
# Mania main board.
4F40: 41 82 -> 48 00

View File

@ -0,0 +1,4 @@
# Skip Dongle Check
B8EC: 48 00 E8 E1 -> 38 60 00 00
B914: 48 00 E7 4D -> 38 60 00 00
B924: 48 00 E1 C9 -> 38 60 00 00

View File

@ -0,0 +1,3 @@
# Skip Dongle Check
CCFC: 48 01 43 F9 -> 38 60 00 00
EFD8: 48 01 21 1D -> 38 60 00 00

View File

@ -0,0 +1,8 @@
# Skip Dongle Check
509B6: FF FF -> 00 00
56498: 48 03 4A 4D -> 38 60 00 00
56520: 48 03 42 E5 -> 38 60 00 00
78A84: 48 01 24 61 -> 38 60 00 00
78ABC: 48 01 1D 49 -> 38 60 00 00
78BD8: 41 82 -> 48 00
8AF4A: FF FF -> 00 00

View File

@ -0,0 +1,6 @@
# Skip Dongle Check
5BEB0: 48 04 20 2D -> 38 60 00 00
5BF54: 48 04 3B 21 -> 38 60 00 00
858E0: 48 01 85 FD -> 38 60 00 00
859FC: 41 82 -> 48 00
8E622: FF FF -> 00 00

View File

@ -0,0 +1,7 @@
# Skip Dongle Check on Mickey Tunes original disk.
B924: 48 01 26 01 -> 38 60 00 00
CD7C: 3D 60 80 10 -> 39 60 00 00
CD80: 81 6B 02 94 -> 39 60 00 00
DB10: 48 01 04 15 -> 38 60 00 00
1D0FE: FF FF -> 00 00
1E66A: FF Ff -> 00 00

View File

@ -0,0 +1,6 @@
# Skip Dongle Check on Mickey Tunes Update disk
30524: 48 01 D0 09 -> 38 60 00 00
31E0C: 3D 60 80 15 81 6B 6A 4C -> 39 60 00 00 39 60 00 00
32BB0: 48 01 A9 7D -> 38 60 00 00
43A76: FF FF -> 00 00
4DC72: FF FF -> 00 00

1
requirements.txt Normal file
View File

@ -0,0 +1 @@
git+https://github.com/DragonMinded/arcadeutils.git@main#egg=arcadeutils

View File

@ -1,113 +0,0 @@
#! /usr/bin/env python3
import argparse
import os
import sys
from binary import Binary
def main() -> int:
# Create the argument parser
parser = argparse.ArgumentParser(
description="Utilities for diffing or patching binary files.",
)
subparsers = parser.add_subparsers(help='commands', dest='command')
# Parser for diffing two binary files
diff_parser = subparsers.add_parser('diff', help='Diff two same-length binary files.')
diff_parser.add_argument(
'file1',
metavar='FILE1',
type=str,
help='The base file that we will output diffs relative to.',
)
diff_parser.add_argument(
'file2',
metavar='FILE2',
type=str,
help='The file that we will compare against the base file to find diffs.',
)
diff_parser.add_argument(
'--patch-file',
metavar='FILE',
type=str,
help='Write patches to a file instead of stdout.',
)
# Parser for patching a binary file
patch_parser = subparsers.add_parser('patch', help='Patch a binary file.')
patch_parser.add_argument(
'bin',
metavar='BIN',
type=str,
help='The binary file we should patch.',
)
patch_parser.add_argument(
'out',
metavar='OUT',
type=str,
help='The file we should write the patched binary to.',
)
patch_parser.add_argument(
'--patch-file',
metavar='FILE',
type=str,
help='Read patches from a file instead of stdin.',
)
patch_parser.add_argument(
'--reverse',
action="store_true",
help='Perform the patch in reverse.',
)
# Grab what we're doing
args = parser.parse_args()
if args.command == 'diff':
with open(args.file1, "rb") as fp:
file1 = fp.read()
with open(args.file2, "rb") as fp:
file2 = fp.read()
try:
differences = Binary.diff(file1, file2)
except Exception as e:
print(f"Could not diff {args.file1} against {args.file2}: {str(e)}", file=sys.stderr)
return 1
if not args.patch_file:
for line in differences:
print(line)
else:
with open(args.patch_file, "w") as fp:
fp.write(os.linesep.join(differences))
elif args.command == 'patch':
with open(args.bin, "rb") as fp:
old = fp.read()
if not args.patch_file:
differences = sys.stdin.readlines()
else:
with open(args.patch_file, "r") as fp:
differences = fp.readlines()
differences = [d.strip() for d in differences if d.strip()]
try:
new = Binary.patch(old, differences, reverse=args.reverse)
except Exception as e:
print(f"Could not patch {args.bin}: {str(e)}", file=sys.stderr)
return 1
with open(args.out, "wb") as fp:
fp.write(new)
print(f"Patched {args.bin} and wrote to {args.out}.")
else:
print("Please specify a valid command!", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@ -1,263 +0,0 @@
from typing import List, Optional, Tuple, cast
from typing_extensions import Final
class BinaryException(Exception):
pass
class Binary:
CHUNK_SIZE: Final[int] = 1024
@staticmethod
def _hex(val: int) -> str:
out = hex(val)[2:]
out = out.upper()
if len(out) == 1:
out = "0" + out
return out
@staticmethod
def diff(bin1: bytes, bin2: bytes) -> List[str]:
binlength = len(bin1)
if binlength != len(bin2):
raise BinaryException("Cannot diff different-sized binary blobs!")
# First, get the list of differences
differences: List[Tuple[int, bytes, bytes]] = []
# Chunk the differences, assuming files are usually about the same,
# for a massive speed boost.
for offset in range(0, binlength, Binary.CHUNK_SIZE):
if bin1[offset:(offset + Binary.CHUNK_SIZE)] != bin2[offset:(offset + Binary.CHUNK_SIZE)]:
for i in range(Binary.CHUNK_SIZE):
byte1 = bin1[offset + i]
byte2 = bin2[offset + i]
if byte1 != byte2:
differences.append((offset + i, bytes([byte1]), bytes([byte2])))
# Don't bother with any combination crap if we have nothing to do
if not differences:
return []
# Now, combine them for easier printing
cur_block: Tuple[int, bytes, bytes] = differences[0]
ret: List[str] = []
# Now, include the original byte size for later comparison/checks
ret.append(f"# File size: {len(bin1)}")
def _hexrun(val: bytes) -> str:
return " ".join(Binary._hex(v) for v in val)
def _output(val: Tuple[int, bytes, bytes]) -> None:
start = val[0] - len(val[1]) + 1
ret.append(
f"{Binary._hex(start)}: {_hexrun(val[1])} -> {_hexrun(val[2])}"
)
def _combine(val: Tuple[int, bytes, bytes]) -> None:
nonlocal cur_block
if cur_block[0] + 1 == val[0]:
# This is a continuation of a run
cur_block = (
val[0],
cur_block[1] + val[1],
cur_block[2] + val[2],
)
else:
# This is a new run
_output(cur_block)
cur_block = val
# Combine and output runs of differences
for diff in differences[1:]:
_combine(diff)
# Make sure we output the last difference
_output(cur_block)
# Return our summation
return ret
@staticmethod
def size(patchlines: List[str]) -> Optional[int]:
for patch in patchlines:
if patch.startswith('#'):
# This is a comment, ignore it, unless its a file-size comment
patch = patch[1:].strip().lower()
if patch.startswith('file size:'):
return int(patch[10:].strip())
return None
@staticmethod
def _convert(val: str) -> Optional[int]:
val = val.strip()
if val == '*':
return None
return int(val, 16)
@staticmethod
def _gather_differences(patchlines: List[str], reverse: bool) -> List[Tuple[int, Optional[bytes], bytes]]:
# First, separate out into a list of offsets and old/new bytes
differences: List[Tuple[int, Optional[bytes], bytes]] = []
for patch in patchlines:
if patch.startswith('#'):
# This is a comment, ignore it.
continue
start_offset, patch_contents = patch.split(':', 1)
before, after = patch_contents.split('->')
beforevals = [
Binary._convert(x) for x in before.split(" ") if x.strip()
]
aftervals = [
Binary._convert(x) for x in after.split(" ") if x.strip()
]
if len(beforevals) != len(aftervals):
raise BinaryException(
f"Patch before and after length mismatch at "
f"offset {start_offset}!"
)
if len(beforevals) == 0:
raise BinaryException(
f"Must have at least one byte to change at "
f"offset {start_offset}!"
)
offset = int(start_offset.strip(), 16)
for i in range(len(beforevals)):
if aftervals[i] is None:
raise BinaryException(
f"Cannot convert a location to a wildcard "
f"at offset {start_offset}"
)
if beforevals[i] is None and reverse:
raise BinaryException(
f"Patch offset {start_offset} specifies a wildcard and cannot "
f"be reversed!"
)
differences.append(
(
offset + i,
bytes([beforevals[i] or 0]) if beforevals[i] is not None else None,
bytes([aftervals[i] or 0]),
)
)
# Now, if we're doing the reverse, just switch them
if reverse:
# We cast here because mypy can't see that we have already asserted that x[2] will never
# be optional in the above loop if reverse is set to True.
differences = [cast(Tuple[int, Optional[bytes], bytes], (x[0], x[2], x[1])) for x in differences]
# Finally, return it
return differences
@staticmethod
def patch(
binary: bytes,
patchlines: List[str],
*,
reverse: bool = False,
) -> bytes:
# First, grab the differences
file_size = Binary.size(patchlines)
if file_size is not None and file_size != len(binary):
raise BinaryException(
f"Patch is for binary of size {file_size} but binary is {len(binary)} "
f"bytes long!"
)
differences: List[Tuple[int, Optional[bytes], bytes]] = sorted(
Binary._gather_differences(patchlines, reverse),
key=lambda diff: diff[0],
)
chunks: List[bytes] = []
last_patch_end: int = 0
# Now, apply the changes to the binary data
for diff in differences:
offset, old, new = diff
if len(binary) < offset:
raise BinaryException(
f"Patch offset {Binary._hex(offset)} is beyond the end of "
f"the binary!"
)
if old is not None and binary[offset:(offset + 1)] != old:
raise BinaryException(
f"Patch offset {Binary._hex(offset)} expecting {Binary._hex(old[0])} "
f"but found {Binary._hex(binary[offset])}!"
)
if last_patch_end < offset:
chunks.append(binary[last_patch_end:offset])
chunks.append(new)
last_patch_end = offset + 1
# Return the new data!
chunks.append(binary[last_patch_end:])
return b"".join(chunks)
@staticmethod
def can_patch(
binary: bytes,
patchlines: List[str],
*,
reverse: bool = False,
ignore_size_differences: bool = False,
) -> Tuple[bool, str]:
# First, grab the differences
if not ignore_size_differences:
file_size = Binary.size(patchlines)
if file_size is not None and file_size != len(binary):
return (
False,
f"Patch is for binary of size {file_size} but binary is {len(binary)} "
f"bytes long!"
)
differences: List[Tuple[int, Optional[bytes], bytes]] = Binary._gather_differences(patchlines, reverse)
# Now, verify the changes to the binary data
for diff in differences:
offset, old, _ = diff
if len(binary) < offset:
return (
False,
f"Patch offset {Binary._hex(offset)} is beyond the end of "
f"the binary!"
)
if old is not None and binary[offset:(offset + 1)] != old:
return (
False,
f"Patch offset {Binary._hex(offset)} expecting {Binary._hex(old[0])} "
f"but found {Binary._hex(binary[offset])}!"
)
# Didn't find any problems
return (True, "")
@staticmethod
def description(patchlines: List[str]) -> Optional[str]:
for patch in patchlines:
if patch.startswith('#'):
# This is a comment, ignore it, unless its a description comment
patch = patch[1:].strip().lower()
if patch.startswith('description:'):
return patch[12:].strip()
return None
@staticmethod
def needed_amount(patchlines: List[str]) -> int:
# First, grab the differences.
differences: List[Tuple[int, Optional[bytes], bytes]] = Binary._gather_differences(patchlines, False)
# Now, get the maximum byte we need to apply this patch.
return max([offset for offset, _, _ in differences]) + 1 if differences else 0

View File

@ -4,7 +4,7 @@ import os
import sys
from typing import Dict, List
from binary import Binary
from arcadeutils.binary import BinaryDiff
from firebeat import FirebeatExe
@ -223,14 +223,14 @@ def main() -> int:
patched = False
unpatched = False
try:
Binary.patch(unpacked, differences, reverse=False)
BinaryDiff.patch(unpacked, differences, reverse=False)
unpatched = True
except Exception:
# It wasn't pristine.
pass
try:
Binary.patch(unpacked, differences, reverse=True)
BinaryDiff.patch(unpacked, differences, reverse=True)
patched = True
except Exception:
# It wasn't patched.

View File

@ -3,7 +3,7 @@ import argparse
import os
import sys
from binary import Binary
from arcadeutils.binary import BinaryDiff
from firebeat import FirebeatExe
@ -151,7 +151,7 @@ def main() -> int:
try:
file1 = FirebeatExe.exe_to_raw(file1, is_ppp=args.ppp)
file2 = FirebeatExe.exe_to_raw(file2, is_ppp=args.ppp)
differences = Binary.diff(file1, file2)
differences = BinaryDiff.diff(file1, file2)
except Exception as e:
print(f"Could not diff {args.file1} against {args.file2}: {str(e)}", file=sys.stderr)
return 1
@ -175,7 +175,7 @@ def main() -> int:
try:
old = FirebeatExe.exe_to_raw(old, is_ppp=args.ppp)
new = Binary.patch(old, differences, reverse=args.reverse)
new = BinaryDiff.patch(old, differences, reverse=args.reverse)
new = FirebeatExe.raw_to_exe(new, is_ppp=args.ppp)
except Exception as e:
print(f"Could not patch {args.exe}: {str(e)}", file=sys.stderr)