importer

package
v0.0.117 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 17, 2026 License: GPL-3.0 Imports: 30 Imported by: 0

Documentation

Overview

Package importer writes conversation history from another chat system — users, channels, direct and group conversations, memberships, messages, threads, reactions, pins, file attachments and custom emoji — straight into ex's store.

Contract: the importer is built only on the storage layers (internal/store, internal/storage, internal/search, internal/cache), never on internal/service. Nothing it does publishes WebSocket events, sends notifications or mobile pushes, bumps alerted-unread badges, records activity, or starts agent runs: imported history arrives silently and already read. Callers supply deterministic IDs (message IDs are ULIDs whose time is the message's original creation time, so imported messages interleave correctly with existing ones), which makes every write idempotent — re-running any call after a crash is safe and never duplicates data.

Ordering: users before anything that references them; channels and conversations before their members and messages; attachments before the messages that carry them; messages of a parent before FinalizeParent and IndexParent for that parent. Calls for different parents are independent and may run concurrently; calls for the same parent must be serialized (PutMessages writes message rows in batches, unconditionally, right after its existence check).

Index

Constants

View Source
const (
	ParentChannel      = "channel"
	ParentConversation = "conversation"
)

Parent types.

View Source
const (
	SystemAuthorID  = "system"
	WebhookAuthorID = "webhook"
)

SystemAuthorID and WebhookAuthorID are the author sentinels ex uses for system notices (join/leave) and incoming-webhook posts. Neither is a user: they never follow threads or keep read state.

Variables

View Source
var ErrInvalidSpec = errors.New("importer: invalid spec")

ErrInvalidSpec wraps every input validation failure.

View Source
var ErrMessageDeleted = errors.New("importer: message is deleted")

ErrMessageDeleted is returned when patching a tombstoned message.

View Source
var ErrNoObjectStore = errors.New("importer: no object store configured")

ErrNoObjectStore is returned by PutAttachment when no object store is set.

Functions

This section is empty.

Types

type AttachmentSpec

type AttachmentSpec struct {
	ID          string
	Filename    string
	ContentType string // defaults to application/octet-stream
	// Size, when > 0, must match the bytes read.
	Size      int64
	Width     int
	Height    int
	CreatedBy string
	CreatedAt time.Time
}

AttachmentSpec describes a file to import.

type AttachmentState

type AttachmentState struct {
	Record      bool // attachment row present
	Object      bool // body present in object storage
	Size        int64
	SHA256      string
	ContentType string
	References  int // messages referencing it
}

AttachmentState says whether an attachment's record and stored body exist.

type AudienceInvalidator added in v0.0.97

type AudienceInvalidator interface {
	ChannelAudiencesChanged(ctx context.Context, channelIDs ...string) error
}

AudienceInvalidator moves the notification audience generations of the channels whose memberships an import changed (cache.RedisCache is one), so every running ex instance alerts the new audience from its very next message instead of a member list it cached before the import.

type Channel

type Channel struct {
	ID       string `json:"id"`
	Slug     string `json:"slug"`
	Name     string `json:"name"`
	Type     string `json:"type"` // "public" | "private"
	Archived bool   `json:"archived"`
}

Channel is an existing ex channel as seen by the importer.

type ChannelSpec

type ChannelSpec struct {
	ID          string
	Slug        string
	Name        string // defaults to Slug
	Description string
	Type        string // "public" | "private"
	CreatedBy   string
	CreatedAt   time.Time
	// Archived creates the channel archived (its members are kept).
	Archived bool
}

ChannelSpec describes a channel to create.

type ConversationSpec

type ConversationSpec struct {
	Type           string // "dm" | "group"
	ParticipantIDs []string
	Name           string // group name (optional)
	CreatedBy      string
	CreatedAt      time.Time
}

ConversationSpec describes a direct (two people, or one for a self-DM) or group conversation.

type Deps

type Deps struct {
	DB        *store.DB
	Objects   ObjectStore
	Search    SearchIndex
	Indexer   search.Indexer
	Cache     UserCache
	Audiences AudienceInvalidator
	TempDir   string
	// contains filtered or unexported fields
}

Deps are the importer's collaborators. DB is required; Objects is required for PutAttachment; Search, Indexer, Cache and Audiences are optional. Without Audiences, running ex instances may keep alerting a channel's pre-import members for as long as they cache an audience (two minutes).

type EmailTakenError

type EmailTakenError struct {
	Email      string
	ExistingID string
}

EmailTakenError reports that the email already belongs to another ex user. Existing users are never modified; map the source user to ExistingID instead.

func (*EmailTakenError) Error

func (e *EmailTakenError) Error() string

type EmojiSpec added in v0.0.105

type EmojiSpec struct {
	Name      string
	CreatedBy string // ex user ID
	CreatedAt time.Time
}

EmojiSpec describes a custom emoji to create.

type FinalizeResult

type FinalizeResult struct {
	ThreadsUpdated int
	// ThreadsMissingRoot counts roots that don't exist in the parent (their
	// replies stay without thread metadata).
	ThreadsMissingRoot int
	FollowsWritten     int
}

FinalizeResult summarizes a FinalizeParent call.

type FinalizeSpec

type FinalizeSpec struct {
	ParentID   string
	ParentType string
	// NewTopLevel is PutResult.TopLevelInserted (summed over the calls).
	NewTopLevel int
	// ThreadRootIDs are the roots that received new replies.
	ThreadRootIDs []string
	// LastActivityAt is the newest imported message's time; a conversation's
	// sidebar position moves forward to it (never backward).
	LastActivityAt time.Time
}

FinalizeSpec describes what PutMessages added to one parent.

type ForeignMessagesError

type ForeignMessagesError struct {
	ParentID string
	Count    int
}

ForeignMessagesError is returned when a channel or conversation holds messages that were not part of the import and removal was not forced.

func (*ForeignMessagesError) Error

func (e *ForeignMessagesError) Error() string

type Importer

type Importer struct {
	// contains filtered or unexported fields
}

Importer writes history into ex. Safe for concurrent use.

func New

func New(d Deps) (*Importer, error)

New builds an importer from explicit collaborators (tests, custom wiring).

func Open

func Open(ctx context.Context, o Options) (*Importer, error)

Open connects to ex's storage with the given settings.

func (*Importer) AddChannelMembers

func (im *Importer) AddChannelMembers(ctx context.Context, channelID string, members []MemberSpec) (added int, err error)

AddChannelMembers adds members to a channel (both the channel-side and the user-side rows). Existing memberships are left untouched; no system message is posted. New members start caught up on the channel's current messages, and are alerted by the channel's next message like any other member. Returns how many memberships were added.

func (*Importer) AttachmentExists

func (im *Importer) AttachmentExists(ctx context.Context, ids []string) (map[string]AttachmentState, error)

AttachmentExists checks attachments' records and bodies (one batched read plus one HEAD per existing record). IDs without a record map to the zero state.

func (*Importer) Channel added in v0.0.99

func (im *Importer) Channel(ctx context.Context, id string) (Channel, error)

Channel reads one ex channel by ID. Unlike Channels it is a single read, cheap enough to check a few channels (e.g. whether one is archived) at any point of an import.

func (*Importer) Channels

func (im *Importer) Channels(ctx context.Context) ([]Channel, error)

Channels lists every ex channel — public and private, archived included.

func (*Importer) Close

func (im *Importer) Close() error

Close releases connections opened by Open.

func (*Importer) EnsureChannel

func (im *Importer) EnsureChannel(ctx context.Context, spec ChannelSpec) (bool, error)

EnsureChannel creates the channel with its historical creation time unless a channel with spec.ID already exists (then it is left as is). Name, slug and description go through ex's channel validators. Returns whether it was created; the stored channel is (re)indexed for search either way.

func (*Importer) EnsureConversation

func (im *Importer) EnsureConversation(ctx context.Context, spec ConversationSpec) (string, bool, error)

EnsureConversation creates the conversation (already activated, so every participant sees it) unless it exists. Its ID is derived from the participants exactly as ex derives it, so an existing conversation between the same people is reused. Returns the ID and whether it was created.

func (*Importer) FinalizeParent

func (im *Importer) FinalizeParent(ctx context.Context, spec FinalizeSpec) (FinalizeResult, error)

FinalizeParent brings a parent's derived state in line with imported messages. Thread roots and replies are read from the parent's partition with strongly consistent reads (one pass for all touched threads), so replies written just before are counted — the thread index (GSI1) can lag behind a bulk write and would leave threads with too low a reply count:

  • each touched thread root gets its reply count, last reply time and recent reply authors recomputed from all its replies;
  • the root author and every reply author follow the thread (an explicit unfollow is never overwritten): the reply path finds a thread's audience in its follow rows;
  • the parent's unread counter and every member's read watermark advance by NewTopLevel together, so imported history is read and existing unread counts don't change;
  • a conversation's activity time moves forward to LastActivityAt.

Calling it twice for the same import advances both counters twice, which leaves unread counts unchanged.

func (*Importer) GetMessages

func (im *Importer) GetMessages(ctx context.Context, parentID string, ids []string) ([]Message, error)

GetMessages returns the given messages of one parent (batched reads). Missing IDs are absent from the result; order is not guaranteed.

func (*Importer) IndexParent

func (im *Importer) IndexParent(ctx context.Context, parentID, parentType string) (int, int, error)

IndexParent writes a parent's messages and the files they reference to the search index (tombstoned, system and no-index messages are skipped). No-op without search. Returns how many message and file documents were written.

func (*Importer) InvalidateUsers

func (im *Importer) InvalidateUsers(ctx context.Context, ids []string) error

InvalidateUsers drops ex's cached records for the given users so the next read sees what was written. No-op without a cache.

func (*Importer) ParentStats

func (im *Importer) ParentStats(ctx context.Context, parentID, parentType string) (ParentStats, error)

ParentStats counts what a parent holds. It walks the parent's whole message partition (one Query page per statsPageSize messages) plus its pin and file index rows and members' read state, so it costs roughly one read unit per 4 KB of messages: use it for validation, not on a hot path.

func (*Importer) PutAttachment

func (im *Importer) PutAttachment(ctx context.Context, spec AttachmentSpec, open func(context.Context) (io.ReadCloser, error)) (bool, error)

PutAttachment stores a file and its attachment record. open is only called when the attachment does not exist yet, so re-runs skip the download. The body is spooled to a temporary file while its SHA-256 is computed, then uploaded in one checksummed PUT — the record is written already verified, so ex serves it without re-reading (image thumbnails are still generated lazily on first view). Returns whether it was created.

func (*Importer) PutMessages

func (im *Importer) PutMessages(ctx context.Context, parentID, parentType string, specs []MessageSpec) (PutResult, error)

PutMessages inserts the messages that don't exist yet into a parent. Messages already present (by ID) are skipped untouched — use UpdateMessage or TombstoneMessage to change them. Every spec is validated before anything is written. The new messages' side rows (attachment references, the parent's file and pin index) are written first and the message rows (with the thread index) last, in batches, so a crash midway leaves nothing a re-run won't complete.

The message rows are written unconditionally after the existence check, so calls for the same parent must not run concurrently (see the package doc).

func (*Importer) RemoveChannel

func (im *Importer) RemoveChannel(ctx context.Context, spec RemoveChannelSpec) (RemoveResult, error)

RemoveChannel deletes a channel the import created: all its messages (as RemoveMessages, without counter updates), memberships on both sides, its slug lock, everything else in its partition and its search document.

func (*Importer) RemoveConversation

func (im *Importer) RemoveConversation(ctx context.Context, spec RemoveConversationSpec) (RemoveResult, error)

RemoveConversation deletes a conversation the import created: its messages (as RemoveMessages, without counter updates), its metadata and member rows, and every participant's sidebar row.

func (*Importer) RemoveMembers

func (im *Importer) RemoveMembers(ctx context.Context, channelID string, userIDs []string) (err error)

RemoveMembers removes users from a channel (both membership rows). Users who aren't members are skipped. The channel's next message no longer alerts them.

func (*Importer) RemoveMessages

func (im *Importer) RemoveMessages(ctx context.Context, parentID, parentType string, ids []string) (RemoveResult, error)

RemoveMessages hard-deletes messages of one parent together with their pin and file index rows, their references on attachments (an attachment no message references any more is deleted with its stored objects), thread follows of removed roots and their search documents. Thread roots that lost replies get their thread metadata recomputed, and the parent's unread counter and every member's read watermark go down by the number of removed top-level, non-system messages (never below zero, and no watermark above the counter), so unread counts stay as they were.

func (*Importer) RemoveUser

func (im *Importer) RemoveUser(ctx context.Context, spec RemoveUserSpec) error

RemoveUser deletes a user the import created: profile, email index, search document and cached record. Memberships are not touched — remove them first (Force only skips the in-use check).

func (*Importer) TombstoneMessage

func (im *Importer) TombstoneMessage(ctx context.Context, parentID, msgID string) error

TombstoneMessage soft-deletes a message the way ex does (the row stays so threads keep their shape; body, attachments, reactions and pin are cleared), drops its attachment references, file and pin index rows, and removes it from search. Already-deleted or unknown messages are a no-op.

func (*Importer) UpdateMessage

func (im *Importer) UpdateMessage(ctx context.Context, parentID, msgID string, p MessagePatch) error

UpdateMessage applies a patch to an existing message, keeping the pin index in step. Returns store.ErrNotFound (wrapped) for an unknown message.

func (*Importer) UpsertEmoji added in v0.0.105

func (im *Importer) UpsertEmoji(ctx context.Context, spec EmojiSpec, img []byte) (bool, error)

UpsertEmoji creates a workspace custom emoji from image unless one with spec.Name exists; an existing emoji is left untouched. The image is checked and kept the way ex keeps an upload: a still picture is shrunk to 256px and re-encoded as lossless WebP, an animated one (GIF, animated WebP) is copied. Messages and reactions refer to custom emoji by name, so history imported with ":name:" shows the emoji once it exists. Connected clients see it after their next reload (no event is published). Returns whether it was created.

func (*Importer) UpsertUser

func (im *Importer) UpsertUser(ctx context.Context, spec UserSpec) (bool, error)

UpsertUser creates a single-sign-on member account (no password), or a bot account for spec.Bot, when no user with spec.ID exists yet. An existing user with that ID is left untouched, except that spec.Bot turns it into a bot account; an existing user with the same email but another ID yields *EmailTakenError. The display name is trimmed and capped like the sign-in flow does. A person is (re)indexed for search either way, a bot is removed from it.

func (*Importer) Users

func (im *Importer) Users(ctx context.Context) ([]User, error)

Users lists every ex user.

type MemberSpec

type MemberSpec struct {
	UserID   string
	Role     model.ChannelRole // 0 = member
	JoinedAt time.Time
	Muted    bool
	Favorite bool
}

MemberSpec describes one channel member.

type Message

type Message struct {
	ID              string
	AuthorID        string
	Body            string
	CreatedAt       time.Time
	EditedAt        *time.Time
	Deleted         bool
	System          bool
	ParentMessageID string
	Reactions       map[string][]string
	Pinned          bool
	AttachmentIDs   []string
	ReplyCount      int
}

Message is the current state of a stored message.

type MessagePatch

type MessagePatch struct {
	Body     *string
	EditedAt *time.Time
	// Reactions, when non-nil, replaces the whole reaction map.
	Reactions map[string][]string
	Pinned    *bool
	PinnedBy  string
	PinnedAt  *time.Time
}

MessagePatch lists the fields an incremental re-sync may change. Nil fields are left as they are.

type MessageSpec

type MessageSpec struct {
	ID              string
	AuthorID        string
	Body            string
	CreatedAt       time.Time
	EditedAt        *time.Time
	ParentMessageID string // thread root, for replies
	AttachmentIDs   []string
	// Reactions maps an emoji key (unicode or ":shortcode:") to user IDs.
	Reactions map[string][]string
	Pinned    bool
	PinnedAt  *time.Time
	PinnedBy  string
	// System marks a system notice (AuthorID should be SystemAuthorID).
	System             bool
	WebhookUsername    string
	WebhookAvatarURL   string
	WebhookIconEmoji   string
	MessageAttachments []model.MessageAttachment
}

MessageSpec describes one message. ID must be a ULID whose timestamp is CreatedAt (to the millisecond): ex orders messages by ID.

type ObjectStore

type ObjectStore interface {
	PutChecksummedObject(ctx context.Context, key, contentType string, body io.ReadSeeker, size int64, sha256Hex string) (string, error)
	HeadObject(ctx context.Context, key string) (bool, error)
	DeleteObject(ctx context.Context, key string) error
}

ObjectStore is the object storage holding attachment bodies: uploads, existence checks (validation) and deletes (rollback).

type Options

type Options struct {
	AWSRegion        string
	DynamoDBTable    string
	DynamoDBEndpoint string

	S3Bucket    string
	S3Region    string
	S3Endpoint  string
	S3AccessKey string
	S3SecretKey string

	OpenSearchURL       string
	OpenSearchAWSRegion string // set to sign requests with SigV4 (AWS-managed domains)

	RedisURL string

	// TempDir holds attachment bodies while they are hashed before upload
	// (empty = the OS temp dir).
	TempDir string
}

Options are the connection settings Open uses. DynamoDBTable and S3Bucket are required; everything else is optional (empty endpoints mean AWS defaults, empty search/Redis settings disable those integrations).

type ParentStats

type ParentStats struct {
	TopLevel int // live, non-system top-level messages
	Replies  int // live thread replies
	System   int // live system messages
	Deleted  int // tombstoned messages
	Pinned   int // pin index rows
	Files    int // file index rows
	Members  int
	// MessageSeq is the parent's unread counter; MembersUnread counts members
	// whose read watermark is behind it.
	MessageSeq    int64
	MembersUnread int
	// ThreadsMismatched counts thread roots whose stored reply count differs
	// from the number of their replies. A deleted reply stays in its thread
	// as a placeholder and keeps counting, as it does in ex (a tombstoned
	// root likewise keeps its count).
	ThreadsMismatched int
}

ParentStats describes a channel or conversation as stored.

type PutResult

type PutResult struct {
	Inserted int
	Existing int
	// TopLevelInserted counts inserted top-level, non-system messages — the
	// number FinalizeParent adds to the unread counters.
	TopLevelInserted int
	// ThreadRootIDs lists the roots that received inserted replies.
	ThreadRootIDs []string
	// LastCreatedAt is the newest inserted message's time (zero if none).
	LastCreatedAt time.Time
}

PutResult summarizes a PutMessages call.

type RemoveChannelSpec

type RemoveChannelSpec struct {
	ChannelID          string
	ImportedMessageIDs []string
	Force              bool
}

RemoveChannelSpec names a channel to remove. ImportedMessageIDs are the messages the import wrote; any other message in the channel makes the call fail with ForeignMessagesError unless Force is set.

type RemoveConversationSpec

type RemoveConversationSpec struct {
	ConversationID     string
	ImportedMessageIDs []string
	Force              bool
}

RemoveConversationSpec names a conversation to remove; see RemoveChannelSpec.

type RemoveResult

type RemoveResult struct {
	Messages    int // message rows deleted
	Attachments int // attachments deleted (no message referenced them any more)
}

RemoveResult summarizes a removal.

type RemoveUserSpec

type RemoveUserSpec struct {
	UserID string
	Force  bool
}

RemoveUserSpec names a user to remove. Without Force the call refuses a user who still belongs to any channel or conversation, or who has signed in.

type SearchIndex

type SearchIndex interface {
	search.BulkWriter
	DeleteDoc(ctx context.Context, index, id string) error
}

SearchIndex is the search integration: per-document upserts for users and channels, bulk writes for a parent's messages and files, and deletes for tombstoned messages.

type SlugTakenError

type SlugTakenError struct {
	Slug       string
	ExistingID string
}

SlugTakenError reports that another channel already holds the slug.

func (*SlugTakenError) Error

func (e *SlugTakenError) Error() string

type User

type User struct {
	ID           string `json:"id"`
	Email        string `json:"email"` // lowercased
	DisplayName  string `json:"displayName"`
	Status       string `json:"status"` // "active" | "deactivated"
	SystemRole   string `json:"systemRole"`
	AuthProvider string `json:"authProvider"`
}

User is an existing ex user as seen by the importer.

type UserCache

type UserCache interface {
	DeleteUsers(ctx context.Context, userIDs []string) error
}

UserCache drops cached user records after users are written.

type UserInUseError

type UserInUseError struct {
	UserID        string
	Channels      int
	Conversations int
	SignedIn      bool
}

UserInUseError is returned when a user still has memberships or has signed in, and removal was not forced.

func (*UserInUseError) Error

func (e *UserInUseError) Error() string

type UserSpec

type UserSpec struct {
	ID          string
	Email       string
	DisplayName string
	Deactivated bool
	CreatedAt   time.Time
	// Bot makes a machine account, like the ones ex creates for incoming
	// webhooks: nobody signs in as it, and it is left out of search, the
	// people directory and directory sync. Email is ignored: a bot gets an
	// undeliverable ID@bots.ex.invalid address.
	Bot bool
}

UserSpec describes a user to create.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址