RS2:317:Cache

From RuneWiki
Revision as of 10:36, 24 May 2021 by Pazaz (talk | contribs) (→‎Extracting idx3)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

The cache is loaded through 6 files: main_file_cache.idx0, main_file_cache.idx1, main_file_cache.idx2, main_file_cache.idx3, main_file_cache.idx4, and main_file_cache.dat.

.idx* files store byte offsets from the .dat file.

.dat is the actual data used by the client.

.idx0 and the data referenced by it is downloaded over HTTP or JAGGRAB (as a fallback) when the client launches.

.idx1-4 is downloaded on-demand via the game server. Necessary files are downloaded before displaying the login screen while extra files can be streamed in at any time. Almost everything can be downloaded before logging in and exploring the world.

Reading idx files

Each idx file uses 6 bytes per entry, describing an offset and entry length in main_file_cache.dat. Data is not stored sequentially and the previous offset must be read before reading the next one.

{
  "fileSize": "3 bytes",
  "sectorAt": "3 bytes"
}

Moving onto the data file now.

Data is stored in 512-byte sectors with an 8-byte header per sector, resulting in 520 bytes per sector.

Seek to sectorAt * 520. Begin reading file number (2 bytes), file part number (2 bytes), sector number (3 bytes), idx number (1 byte).

{
  "fileNumber": "2 bytes",
  "filePartNumber": "2 bytes",
  "sectorNumber": "3 bytes",
  "cacheNumber": "byte"
}

Take a slice of the next 512 bytes, or remaining file size. Whatever is less.

Repeat the same read as above by seeking to sectorNumber * 520 until your buffer is as long as fileSize.

Extracting idx0

I'm going to refer to each entry in this index as an archive. An archive can have many files inside it.

This index stores references to archives.

Archives are bzip2 compressed data (minus the header "BZh1"). They can be compressed as a whole, or each file inside separately.

There are 8 archives, conventionally named: title.jag, config.jag, interface.jag, media.jag, versionlist.jag, textures.jag, wordenc.jag, and sounds.jag. They have no real extension.

Files inside each archive have a 4-byte name hash that gets used as their identifier. I will refer to files here by their known name hash.

The first 6 bytes of an archive describe the compression: 3 bytes for the decompressed size and 3 bytes for the compressed size.

If decompressed != compressed, the archive is compressed as a whole and must be decompressed before continuing. Prepend the bzip2 header from above.

If decompressed == compressed, the archive is not compressed as a whole and each file is individually compressed. This can be done at a later step.

The next 2 bytes are the total amount of files in the archive (fileCount).

Next up is the file headers. After reading all file headers, the file data begins.

Loop by fileCount.

Read 4 bytes for the name hash.

Read 3 bytes for the decompressed file size.

Read 3 bytes for the compressed file size.

let fileHeaders = [];
for (let i = 0; i < fileCount; ++i) {
  let nameHash = read4Bytes();
  let decompressedSize = read3Bytes();
  let compressedSize = read3Bytes();
  fileHeaders.push({
    nameHash,
    decompressedSize,
    compressedSize,
  });
}

Onto file data.

Loop by fileCount again.

Slice the archive buffer from the current position to fileHeaders[i].compressedSize.

If the archive is not compressed as a whole, now decompress the file. Otherwise, the slice you just made is the file data.

All done! Continue with the next few sections to see how to read the files now.

Image Archives

logo.dat extracted from title.jag

The image-oriented archives are title.jag, media.jag, textures.jag and can all be read the same way:

index.dat describes groups of images. A group can contain a single image or multiple images.

<texture>.dat has the image data. The first two bytes of this file points to an offset in index.dat. Read the image group in index.dat to know how many bytes to later read and how to convert the image data using the color palette.

Image data in this format gets stored as single bytes that refer to a specific palette entry from the image group.

Each image group has a max width (2 bytes), max height (2 bytes), the amount of colors (1 byte), and a color palette (3 bytes per) which results in 255 unique RGB colors. Immediately following this is the image metadata. Read 3-byte integers for paletteCount - 1 loops. Always make the first entry 0 then begin reading.

{
  "maxWidth": "2 bytes",
  "maxHeight": "2 bytes",
  "paletteCount": "byte",
  "palette": [ 0, "3 bytes" ]
}

Each image in a group has an X/Y offset (byte, byte), width (2 bytes), height (2 bytes), and defines a pack type (1 byte) to describe how it was saved. If there are additional images in a group they immediately follow this as well using the same format.

{
  "xOffset": "byte",
  "yOffset": "byte",
  "width": "2 bytes",
  "height": "2 bytes",
  "packType": "byte"
}

Read (width * height) bytes from <texture>.dat starting at offset 2 (after the index.dat offset we read earlier).

Convert this image data into any format you desire.

Pack Type 0

A raw stream of pixels.

for (let i = 0; i < width * height; ++i) {
  pixels[i] = palette[readNextByte()];
}
Pack Type 1

An x, y stream of pixels.

for (let x = 0; x < width; ++x) {
  for (let y = 0; y < height; ++y) {
    pixels[x + y * width] = palette[readNextByte()];
  }
}

Config Archive

Properties for items, floors, NPCs, etc.

Interface Archive

Specifications for all GUI elements.

Version List Archive

Checksums for all data.

Word Enc Archive

Contains lists of bad words. They look like messed up text files but are not read 1:1.

Sounds Archive

.wav files perhaps? TODO.

Extracting idx1

This index stores references to models.

Each file is gzipped. File format TODO.

Extracting idx2

This index stores references to animations.

Each file is gzipped. File format TODO.

Extracting idx3

Scape Main (entry 0) decompressed

This index stores references to music.

Each file is a gzipped .midi file and can be played after decompressing.

Extracting idx4

This index stores references to map data.

Each file is gzipped. File format TODO.