RuneWiki Difference between revisions of "Old Engine/File Store"

Difference between revisions of "Old Engine/File Store"

From RuneWiki
Line 1: Line 1:
This file format, .dat/.idx0-4, was introduced in [https://runescape.fandom.com/wiki/Update:New_game_update_system revision 234]. It was replaced by .dat2/idx0-255 after [https://runescape.fandom.com/wiki/Update:Game_engine_upgraded! revision 400]. It allowed Jagex to push updates without forcing clients to redownload everything compared to using the original [[Old Engine/File Archive|File Archive]] format alone. File archives were inside of .idx0 when this format was used.
This cache format is a sector-based virtual file system. It was used from [https://runescape.fandom.com/wiki/Update:New_game_update_system revision 234] (2004) to [https://runescape.fandom.com/wiki/Update:Game_engine_upgraded! revision 402] (2006). Before revision 234, [[Old Engine/File Archive|File Archives]] were used instead.


.idx files are essentially lookup tables for the .dat store.
It uses two file extensions: dat and idx. Idx should have the store ID following it, i.e. idx0. They represent “Data” and “Index,” the index is a lookup table for each entry with the minimum data necessary to begin reading an entry.


A cache entry uses 6 bytes in the respective idx file:
'''The three main advantages of this format are:'''


*3 bytes for the file size
1) The client can incrementally update its local cache without having to re-download the entire cache each time.


*3 bytes for the starting dat sector
2) The client can load files on-demand, as well as preload most files ahead of time.


A dat sector can be up to 520 bytes long, 8 bytes for the header and the remaining bytes for the cache entry data.
3) Maps and midis can be saved to the cache instead of being streamed from the server.


A sector header looks like:
It has an artificial size limit of 52.8 MB for main_file_cache.dat, the technical limit is around ~8.7GB. This value is calculated by taking the maximum unsigned value of a 24-bit integer and multiplying it by 520 (the size of a data sector).


*2 bytes defining the file ID
Each entry also has an artificial size limit that was increased as time went on. In 317 this limitation was 500 KB, by 377 the limitation was raised to 600 KB.
*2 bytes defining the file part (how many have been read)
*3 bytes defining the next sector (seek here)
*1 byte defining the sector store (.idx file used)


Example:<syntaxhighlight lang="javascript">
This format does not have the ability to store file names so everything is typically referenced using a numerical identifier.
 
== Format ==
The index file uses 6 bytes per entry:
 
* 3 bytes for the file size
 
* 3 bytes for the starting data sector to begin reading at
 
Data sectors are aligned to 520-byte boundaries. The beginning 8 bytes are used as the sector header and the following 512 bytes will be the data for that sector. The last sector is not guaranteed to have a full 512 bytes and the format relies on the remaining file size here instead.
 
The sector header uses 8 bytes per sector:
 
* 2 bytes for the file ID
 
* 2 bytes for the current file part
 
* 3 bytes for the next sector to continue reading at
 
* 1 byte for the store ID
 
== Example ==
<syntaxhighlight lang="javascript">
async read(file) {
async read(file) {
let temp = new Uint8Array(520);
let temp = new Uint8Array(520);

Revision as of 00:47, 8 September 2021

This cache format is a sector-based virtual file system. It was used from revision 234 (2004) to revision 402 (2006). Before revision 234, File Archives were used instead.

It uses two file extensions: dat and idx. Idx should have the store ID following it, i.e. idx0. They represent “Data” and “Index,” the index is a lookup table for each entry with the minimum data necessary to begin reading an entry.

The three main advantages of this format are:

1) The client can incrementally update its local cache without having to re-download the entire cache each time.

2) The client can load files on-demand, as well as preload most files ahead of time.

3) Maps and midis can be saved to the cache instead of being streamed from the server.

It has an artificial size limit of 52.8 MB for main_file_cache.dat, the technical limit is around ~8.7GB. This value is calculated by taking the maximum unsigned value of a 24-bit integer and multiplying it by 520 (the size of a data sector).

Each entry also has an artificial size limit that was increased as time went on. In 317 this limitation was 500 KB, by 377 the limitation was raised to 600 KB.

This format does not have the ability to store file names so everything is typically referenced using a numerical identifier.

Format

The index file uses 6 bytes per entry:

  • 3 bytes for the file size
  • 3 bytes for the starting data sector to begin reading at

Data sectors are aligned to 520-byte boundaries. The beginning 8 bytes are used as the sector header and the following 512 bytes will be the data for that sector. The last sector is not guaranteed to have a full 512 bytes and the format relies on the remaining file size here instead.

The sector header uses 8 bytes per sector:

  • 2 bytes for the file ID
  • 2 bytes for the current file part
  • 3 bytes for the next sector to continue reading at
  • 1 byte for the store ID

Example

async read(file) {
	let temp = new Uint8Array(520);

	this.idx.seek(file * 6);
	await this.idx.read(temp, 0, 6);

	let size = ((temp[0] & 0xFF) << 16) + ((temp[1] & 0xFF) << 8) + (temp[2] & 0xFF);
	let sector = ((temp[3] & 0xFF) << 16) + ((temp[4] & 0xFF) << 8) + (temp[5] & 0xFF);

	if (size > this.maxFileSize) {
		return new Uint8Array();
	}

	if (sector <= 0 || sector > await this.dat.length() / 520) {
		return new Uint8Array();
	}

	let data = new Uint8Array(size);
	let position = 0;

	for (let part = 0; position < size; part++) {
		if (sector === 0) {
			return new Uint8Array();
		}

		let available = size - position;
		if (available > 512) {
			available = 512;
		}

		this.dat.seek(sector * 520);
		await this.dat.read(temp, 0, available + 8);

		let sectorFile = ((temp[0] & 0xFF) << 8) + (temp[1] & 0xFF);
		let sectorPart = ((temp[2] & 0xFF) << 8) + (temp[3] & 0xFF);
		let nextSector = ((temp[4] & 0xFF) << 16) + ((temp[5] & 0xFF) << 8) + (temp[6] & 0xFF);
		let sectorStore = temp[7] & 0xFF;

		if (sectorFile !== file || sectorPart !== part || sectorStore !== this.store) {
			return new Uint8Array();
		}

		if (nextSector < 0 || nextSector > await this.dat.length() / 520) {
			return new Uint8Array();
		}

		for (let i = 0; i < available; i++) {
			data[position++] = temp[i + 8];
		}

		sector = nextSector;
	}
	return data;
}