moderator-toolbox-nxg-for-reddit / util/wiki/schemas/usernotes/sharding

sharding¶

Functions¶

hashUsername()¶

hashUsername(name): number

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:79

Hashes a username to its uint32 shard key with 32-bit FNV-1a over the lowercased name. Lowercased because storage may key one user under both a lowercase and a canonical-cased name (merged by getUser) - both casings must land in the same shard. The function is part of the storage format: changing it strands existing users in the wrong shards.

Parameters¶

name¶

string

Returns¶

number

initialManifest()¶

initialManifest(types): UsernotesManifest

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:133

Builds the manifest for a freshly sharded subreddit: one shard covering the whole hash space.

Parameters¶

types¶

UserNoteColor[]

Returns¶

UsernotesManifest

isUsernotesManifest()¶

isUsernotesManifest(value): value is UsernotesManifest

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:94

Structurally validates a parsed NXG usernotes page as a manifest, including the shard range invariants (sorted, first range starts at 0, no duplicate starts or pages).

Parameters¶

value¶

unknown

Returns¶

value is UsernotesManifest

mergeShardData()¶

mergeShardData(shards, types): UserNotesData

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:214

Merges decoded shard datasets into one in-memory dataset. Users colliding across shards should be impossible (ranges are disjoint); if it happens anyway the last shard wins and the result is flagged corrupted.

Parameters¶

shards¶

UserNotesData[]

The decoded per-shard datasets, in manifest order.

types¶

UserNoteColor[]

The manifest’s type definitions, attached to the result.

Returns¶

UserNotesData

partitionUsers()¶

partitionUsers(users, manifest): Map<string, Record<string, UsernotesUser>>

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:161

Partitions a user map into per-shard user maps by username hash. Every shard in the manifest gets an entry, even when empty - an emptied shard is rewritten as a valid empty envelope rather than removed (ranges never merge).

Parameters¶

users¶

Record<string, UsernotesUser>

manifest¶

UsernotesManifest

Returns¶

Map<string, Record<string, UsernotesUser>>

pickSplitBoundary()¶

pickSplitBoundary(users, rangeStart): number | null

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:185

Picks the hash boundary to split an overflowing shard’s range at: the median user’s hash, nudged forward past duplicates so the boundary is strictly inside the range and both halves are non-empty.

Parameters¶

users¶

UsernotesUser[]

The users currently in the shard.

rangeStart¶

number

The shard’s inclusive lower hash bound.

Returns¶

number | null

The new boundary (start of the upper half), or null when the shard cannot split (fewer than two distinct hashes above rangeStart).

shardIndexForHash()¶

shardIndexForHash(manifest, hash): number

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:144

Returns the index of the manifest shard whose range contains hash (binary search on starts).

Parameters¶

manifest¶

UsernotesManifest

hash¶

number

Returns¶

number

shardPageName()¶

shardPageName(gen, start): string

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:128

Builds the page-name suffix for a shard created at gen covering hashes from start.

Parameters¶

gen¶

number

start¶

number

Returns¶

string

Interfaces¶

UsernotesManifest¶

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:52

The JSON manifest stored at the NXG usernotes page.

Properties¶

format¶

format: "tbun-manifest"

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:53

gen¶

gen: number

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:56

Monotonic generation counter, bumped whenever the shard list changes. Makes split page names unique.

retired?¶

optional retired?: string[]

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:69

Pages retired by a split whose best-effort tombstone write failed; retried on later manifest writes and dropped once tombstoned.

shards¶

shards: UsernotesShardRef[]

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:64

Shard ranges, sorted by start; shards[0].start is always 0.

types¶

types: UserNoteColor[]

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:62

Usernote type definitions, making the sharded layout self-contained (v6 envelopes only carry type keys). Seeded from the subreddit config’s usernoteColors on write.

ver¶

ver: 7

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:54

UsernotesShardRef¶

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:41

One shard’s entry in the manifest.

Properties¶

page¶

page: string

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:48

Page-name suffix under the NXG usernotes prefix, e.g. s3-80000000.

start¶

start: number

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:46

Inclusive uint32 lower bound of the shard’s hash range. Shard i covers [start_i, start_{i+1}); the last shard covers through 2^32 - 1.

Variables¶

SHARD_HARD_LIMIT_BYTES¶

const SHARD_HARD_LIMIT_BYTES: 510000 = 510_000

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:38

Absolute ceiling for a shard that cannot split (all of its users share one hash - in practice, a single user with enormous notes). Past this the save fails rather than risk a 413 from Reddit.

SHARD_SOFT_LIMIT_BYTES¶

const SHARD_SOFT_LIMIT_BYTES: 480000 = 480_000

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:31

Soft per-shard limit on the posted page bytes; a dirty shard whose encoded envelope exceeds this is split. Sits safely under Reddit’s 512KB cap so a shard can absorb further growth between saves.

USERNOTES_MANIFEST_FORMAT¶

const USERNOTES_MANIFEST_FORMAT: "tbun-manifest" = 'tbun-manifest'

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:18

Manifest format marker; identifies the page as a shard manifest.

USERNOTES_MANIFEST_VER¶

const USERNOTES_MANIFEST_VER: 7 = 7

Defined in: extension/data/util/wiki/schemas/usernotes/sharding.ts:24

Current manifest schema version. Continues the legacy usernotes numbering: v6 is the single-page zlib format, v7 the sharded manifest layout.