Old Engine/File Store

This cache format is a sector-based virtual file system. It was used from revision 234 (2004) to revision 402 (2006). Before revision 234, File Archives were used instead.

It uses two file extensions: dat and idx. Idx should have the store ID following it, i.e. idx0. They represent “Data” and “Index,” the index is a lookup table for each entry with the minimum data necessary to begin reading an entry.

The three main advantages of this format are:

1) The client can incrementally update its local cache without having to re-download the entire cache each time.

2) The client can load files on-demand, as well as preload most files ahead of time.

3) Maps and midis can be saved to the cache instead of being streamed from the server.

It has an artificial size limit of 52.8 MB for main_file_cache.dat, the technical limit is around ~8.7GB. This value is calculated by taking the maximum unsigned value of a 24-bit integer and multiplying it by 520 (the size of a data sector).

Each entry also has an artificial size limit that was increased as time went on. In 317 this limitation was 500 KB, by 377 the limitation was raised to 600 KB.

This format does not have the ability to store file names so everything is typically referenced using a numerical identifier.

Format

The index file uses 6 bytes per entry:

• 3 bytes for the file size

• 3 bytes for the starting data sector to begin reading at

Data sectors are aligned to 520-byte boundaries. The beginning 8 bytes are used as the sector header and the following 512 bytes will be the data for that sector. The last sector is not guaranteed to have a full 512 bytes and the format relies on the remaining file size here instead.

The sector header uses 8 bytes per sector:

• 2 bytes for the file ID

• 2 bytes for the current file part

• 3 bytes for the next sector to continue reading at

• 1 byte for the store ID

Example

async read(file) {
	let temp = new Uint8Array(520);

	this.idx.seek(file * 6);
	await this.idx.read(temp, 0, 6);

	let size = ((temp[0] & 0xFF) << 16) + ((temp[1] & 0xFF) << 8) + (temp[2] & 0xFF);
	let sector = ((temp[3] & 0xFF) << 16) + ((temp[4] & 0xFF) << 8) + (temp[5] & 0xFF);

	if (size > this.maxFileSize) {
		return new Uint8Array();
	}

	if (sector <= 0 || sector > await this.dat.length() / 520) {
		return new Uint8Array();
	}

	let data = new Uint8Array(size);
	let position = 0;

	for (let part = 0; position < size; part++) {
		if (sector === 0) {
			return new Uint8Array();
		}

		let available = size - position;
		if (available > 512) {
			available = 512;
		}

		this.dat.seek(sector * 520);
		await this.dat.read(temp, 0, available + 8);

		let sectorFile = ((temp[0] & 0xFF) << 8) + (temp[1] & 0xFF);
		let sectorPart = ((temp[2] & 0xFF) << 8) + (temp[3] & 0xFF);
		let nextSector = ((temp[4] & 0xFF) << 16) + ((temp[5] & 0xFF) << 8) + (temp[6] & 0xFF);
		let sectorStore = temp[7] & 0xFF;

		if (sectorFile !== file || sectorPart !== part || sectorStore !== this.store) {
			return new Uint8Array();
		}

		if (nextSector < 0 || nextSector > await this.dat.length() / 520) {
			return new Uint8Array();
		}

		for (let i = 0; i < available; i++) {
			data[position++] = temp[i + 8];
		}

		sector = nextSector;
	}
	return data;
}