Actions

Old Engine/File Store: Difference between revisions

From RuneWiki

(Created blank page)
 
No edit summary
 
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
This cache format is a sector-based virtual file system. It was used from [https://runescape.fandom.com/wiki/Update:New_game_update_system revision 234] (2004) to [https://runescape.fandom.com/wiki/Update:Game_engine_upgraded! revision 402] (2006). Before revision 234, [[Old Engine/File Archive|File Archives]] were used instead. See [[Jagex Store]] for the next iteration of the cache format.


It uses two file extensions: dat and idx. Idx should have the store ID following it, i.e. idx0. They represent “Data” and “Index,” the index is a lookup table for each entry with the minimum data necessary to begin reading an entry.
'''The three main advantages of this format are:'''
1) The client can incrementally update its local cache without having to re-download the entire cache each time.
2) The client can load files on-demand, as well as preload most files ahead of time.
3) Maps and midis can be saved to the cache instead of being streamed from the server.
It has an artificial size limit of 52.8 MB for main_file_cache.dat, the technical limit is around ~8.7GB. This value is calculated by taking the maximum unsigned value of a 24-bit integer and multiplying it by 520 (the size of a data sector).
Each entry also has an artificial size limit that was increased as time went on. In 317 this limitation was 500 KB, by 377 the limitation was raised to 600 KB.
This format does not have the ability to store file names so everything is typically referenced using a numerical identifier.
==Format==
'''The index file uses 6 bytes per entry:'''
*3 bytes for the entry size
*3 bytes for the starting data sector to begin reading at
Data sectors are aligned to 520-byte boundaries. The beginning 8 bytes are used as the sector header and the following 512 bytes will be the data for that sector. The last sector is not guaranteed to have a full 512 bytes and the format relies on the remaining entry size here instead.
'''A sector header uses 8 bytes per sector:'''
*2 bytes for the entry ID
*2 bytes for the current entry part
*3 bytes for the next sector to continue reading at
*1 byte for the store ID
==Example==
<syntaxhighlight lang="javascript">
read(index = 0, entry = 0, extract = false) {
    this.idx[index].front().seek(entry * 6);
    let header = this.idx[index].read(6);
    let entrySize = header.readSWord();
    let sectorAt = header.readSWord();
    let data = new ByteBuffer(new Uint8Array(entrySize));
    while (data.available) {
        this.dat.front().seek(sectorAt * 520);
        let sector = this.dat.read(8);
        // todo: sanity checking entryId, currentPart, storeId...
        let entryId = sector.readWord();
        let currentPart = sector.readWord();
        sectorAt = sector.readSWord();
        let storeId = sector.readByte();
        let remaining = 512;
        if (data.available < 512) {
            remaining = data.available;
        }
        data.write(this.dat.read(remaining).raw);
    }
    if (extract && index > 0) {
        return new ByteBuffer(zlib.gunzipSync(data.raw)).raw;
    }
    return data.raw;
}
</syntaxhighlight>

Latest revision as of 21:31, 23 January 2022

This cache format is a sector-based virtual file system. It was used from revision 234 (2004) to revision 402 (2006). Before revision 234, File Archives were used instead. See Jagex Store for the next iteration of the cache format.

It uses two file extensions: dat and idx. Idx should have the store ID following it, i.e. idx0. They represent “Data” and “Index,” the index is a lookup table for each entry with the minimum data necessary to begin reading an entry.

The three main advantages of this format are:

1) The client can incrementally update its local cache without having to re-download the entire cache each time.

2) The client can load files on-demand, as well as preload most files ahead of time.

3) Maps and midis can be saved to the cache instead of being streamed from the server.

It has an artificial size limit of 52.8 MB for main_file_cache.dat, the technical limit is around ~8.7GB. This value is calculated by taking the maximum unsigned value of a 24-bit integer and multiplying it by 520 (the size of a data sector).

Each entry also has an artificial size limit that was increased as time went on. In 317 this limitation was 500 KB, by 377 the limitation was raised to 600 KB.

This format does not have the ability to store file names so everything is typically referenced using a numerical identifier.

Format

The index file uses 6 bytes per entry:

  • 3 bytes for the entry size
  • 3 bytes for the starting data sector to begin reading at

Data sectors are aligned to 520-byte boundaries. The beginning 8 bytes are used as the sector header and the following 512 bytes will be the data for that sector. The last sector is not guaranteed to have a full 512 bytes and the format relies on the remaining entry size here instead.

A sector header uses 8 bytes per sector:

  • 2 bytes for the entry ID
  • 2 bytes for the current entry part
  • 3 bytes for the next sector to continue reading at
  • 1 byte for the store ID

Example

read(index = 0, entry = 0, extract = false) {
    this.idx[index].front().seek(entry * 6);
    let header = this.idx[index].read(6);
    let entrySize = header.readSWord();
    let sectorAt = header.readSWord();

    let data = new ByteBuffer(new Uint8Array(entrySize));

    while (data.available) {
        this.dat.front().seek(sectorAt * 520);
        let sector = this.dat.read(8);
        // todo: sanity checking entryId, currentPart, storeId...
        let entryId = sector.readWord();
        let currentPart = sector.readWord();
        sectorAt = sector.readSWord();
        let storeId = sector.readByte();
        let remaining = 512;
        if (data.available < 512) {
            remaining = data.available;
        }
        data.write(this.dat.read(remaining).raw);
    }

    if (extract && index > 0) {
        return new ByteBuffer(zlib.gunzipSync(data.raw)).raw;
    }

    return data.raw;
}