Documentation
¶
Overview ¶
Package importer writes conversation history from another chat system — users, channels, direct and group conversations, memberships, messages, threads, reactions, pins, file attachments and custom emoji — straight into ex's store.
Contract: the importer is built only on the storage layers (internal/store, internal/storage, internal/search, internal/cache), never on internal/service. Nothing it does publishes WebSocket events, sends notifications or mobile pushes, bumps alerted-unread badges, records activity, or starts agent runs: imported history arrives silently and already read. Callers supply deterministic IDs (message IDs are ULIDs whose time is the message's original creation time, so imported messages interleave correctly with existing ones), which makes every write idempotent — re-running any call after a crash is safe and never duplicates data.
Ordering: users before anything that references them; channels and conversations before their members and messages; attachments before the messages that carry them; messages of a parent before FinalizeParent and IndexParent for that parent. Calls for different parents are independent and may run concurrently; calls for the same parent must be serialized (PutMessages writes message rows in batches, unconditionally, right after its existence check).
Index ¶
- Constants
- Variables
- type AttachmentSpec
- type AttachmentState
- type AudienceInvalidator
- type Channel
- type ChannelSpec
- type ConversationSpec
- type Deps
- type EmailTakenError
- type EmojiSpec
- type FinalizeResult
- type FinalizeSpec
- type ForeignMessagesError
- type Importer
- func (im *Importer) AddChannelMembers(ctx context.Context, channelID string, members []MemberSpec) (added int, err error)
- func (im *Importer) AttachmentExists(ctx context.Context, ids []string) (map[string]AttachmentState, error)
- func (im *Importer) Channel(ctx context.Context, id string) (Channel, error)
- func (im *Importer) Channels(ctx context.Context) ([]Channel, error)
- func (im *Importer) Close() error
- func (im *Importer) EnsureChannel(ctx context.Context, spec ChannelSpec) (bool, error)
- func (im *Importer) EnsureConversation(ctx context.Context, spec ConversationSpec) (string, bool, error)
- func (im *Importer) FinalizeParent(ctx context.Context, spec FinalizeSpec) (FinalizeResult, error)
- func (im *Importer) GetMessages(ctx context.Context, parentID string, ids []string) ([]Message, error)
- func (im *Importer) IndexParent(ctx context.Context, parentID, parentType string) (int, int, error)
- func (im *Importer) InvalidateUsers(ctx context.Context, ids []string) error
- func (im *Importer) ParentStats(ctx context.Context, parentID, parentType string) (ParentStats, error)
- func (im *Importer) PutAttachment(ctx context.Context, spec AttachmentSpec, ...) (bool, error)
- func (im *Importer) PutMessages(ctx context.Context, parentID, parentType string, specs []MessageSpec) (PutResult, error)
- func (im *Importer) RemoveChannel(ctx context.Context, spec RemoveChannelSpec) (RemoveResult, error)
- func (im *Importer) RemoveConversation(ctx context.Context, spec RemoveConversationSpec) (RemoveResult, error)
- func (im *Importer) RemoveMembers(ctx context.Context, channelID string, userIDs []string) (err error)
- func (im *Importer) RemoveMessages(ctx context.Context, parentID, parentType string, ids []string) (RemoveResult, error)
- func (im *Importer) RemoveUser(ctx context.Context, spec RemoveUserSpec) error
- func (im *Importer) TombstoneMessage(ctx context.Context, parentID, msgID string) error
- func (im *Importer) UpdateMessage(ctx context.Context, parentID, msgID string, p MessagePatch) error
- func (im *Importer) UpsertEmoji(ctx context.Context, spec EmojiSpec, img []byte) (bool, error)
- func (im *Importer) UpsertUser(ctx context.Context, spec UserSpec) (bool, error)
- func (im *Importer) Users(ctx context.Context) ([]User, error)
- type MemberSpec
- type Message
- type MessagePatch
- type MessageSpec
- type ObjectStore
- type Options
- type ParentStats
- type PutResult
- type RemoveChannelSpec
- type RemoveConversationSpec
- type RemoveResult
- type RemoveUserSpec
- type SearchIndex
- type SlugTakenError
- type User
- type UserCache
- type UserInUseError
- type UserSpec
Constants ¶
const ( ParentChannel = "channel" ParentConversation = "conversation" )
Parent types.
const ( SystemAuthorID = "system" WebhookAuthorID = "webhook" )
SystemAuthorID and WebhookAuthorID are the author sentinels ex uses for system notices (join/leave) and incoming-webhook posts. Neither is a user: they never follow threads or keep read state.
Variables ¶
var ErrInvalidSpec = errors.New("importer: invalid spec")
ErrInvalidSpec wraps every input validation failure.
var ErrMessageDeleted = errors.New("importer: message is deleted")
ErrMessageDeleted is returned when patching a tombstoned message.
var ErrNoObjectStore = errors.New("importer: no object store configured")
ErrNoObjectStore is returned by PutAttachment when no object store is set.
Functions ¶
This section is empty.
Types ¶
type AttachmentSpec ¶
type AttachmentSpec struct {
ID string
Filename string
ContentType string // defaults to application/octet-stream
// Size, when > 0, must match the bytes read.
Size int64
Width int
Height int
CreatedBy string
CreatedAt time.Time
}
AttachmentSpec describes a file to import.
type AttachmentState ¶
type AttachmentState struct {
Record bool // attachment row present
Object bool // body present in object storage
Size int64
SHA256 string
ContentType string
References int // messages referencing it
}
AttachmentState says whether an attachment's record and stored body exist.
type AudienceInvalidator ¶ added in v0.0.97
type AudienceInvalidator interface {
ChannelAudiencesChanged(ctx context.Context, channelIDs ...string) error
}
AudienceInvalidator moves the notification audience generations of the channels whose memberships an import changed (cache.RedisCache is one), so every running ex instance alerts the new audience from its very next message instead of a member list it cached before the import.
type Channel ¶
type Channel struct {
ID string `json:"id"`
Slug string `json:"slug"`
Name string `json:"name"`
Type string `json:"type"` // "public" | "private"
Archived bool `json:"archived"`
}
Channel is an existing ex channel as seen by the importer.
type ChannelSpec ¶
type ChannelSpec struct {
ID string
Slug string
Name string // defaults to Slug
Description string
Type string // "public" | "private"
CreatedBy string
CreatedAt time.Time
// Archived creates the channel archived (its members are kept).
Archived bool
}
ChannelSpec describes a channel to create.
type ConversationSpec ¶
type ConversationSpec struct {
Type string // "dm" | "group"
ParticipantIDs []string
Name string // group name (optional)
CreatedBy string
CreatedAt time.Time
}
ConversationSpec describes a direct (two people, or one for a self-DM) or group conversation.
type Deps ¶
type Deps struct {
DB *store.DB
Objects ObjectStore
Search SearchIndex
Indexer search.Indexer
Cache UserCache
Audiences AudienceInvalidator
TempDir string
// contains filtered or unexported fields
}
Deps are the importer's collaborators. DB is required; Objects is required for PutAttachment; Search, Indexer, Cache and Audiences are optional. Without Audiences, running ex instances may keep alerting a channel's pre-import members for as long as they cache an audience (two minutes).
type EmailTakenError ¶
EmailTakenError reports that the email already belongs to another ex user. Existing users are never modified; map the source user to ExistingID instead.
func (*EmailTakenError) Error ¶
func (e *EmailTakenError) Error() string
type FinalizeResult ¶
type FinalizeResult struct {
ThreadsUpdated int
// ThreadsMissingRoot counts roots that don't exist in the parent (their
// replies stay without thread metadata).
ThreadsMissingRoot int
FollowsWritten int
}
FinalizeResult summarizes a FinalizeParent call.
type FinalizeSpec ¶
type FinalizeSpec struct {
ParentID string
ParentType string
// NewTopLevel is PutResult.TopLevelInserted (summed over the calls).
NewTopLevel int
// ThreadRootIDs are the roots that received new replies.
ThreadRootIDs []string
// LastActivityAt is the newest imported message's time; a conversation's
// sidebar position moves forward to it (never backward).
LastActivityAt time.Time
}
FinalizeSpec describes what PutMessages added to one parent.
type ForeignMessagesError ¶
ForeignMessagesError is returned when a channel or conversation holds messages that were not part of the import and removal was not forced.
func (*ForeignMessagesError) Error ¶
func (e *ForeignMessagesError) Error() string
type Importer ¶
type Importer struct {
// contains filtered or unexported fields
}
Importer writes history into ex. Safe for concurrent use.
func (*Importer) AddChannelMembers ¶
func (im *Importer) AddChannelMembers(ctx context.Context, channelID string, members []MemberSpec) (added int, err error)
AddChannelMembers adds members to a channel (both the channel-side and the user-side rows). Existing memberships are left untouched; no system message is posted. New members start caught up on the channel's current messages, and are alerted by the channel's next message like any other member. Returns how many memberships were added.
func (*Importer) AttachmentExists ¶
func (im *Importer) AttachmentExists(ctx context.Context, ids []string) (map[string]AttachmentState, error)
AttachmentExists checks attachments' records and bodies (one batched read plus one HEAD per existing record). IDs without a record map to the zero state.
func (*Importer) Channel ¶ added in v0.0.99
Channel reads one ex channel by ID. Unlike Channels it is a single read, cheap enough to check a few channels (e.g. whether one is archived) at any point of an import.
func (*Importer) Channels ¶
Channels lists every ex channel — public and private, archived included.
func (*Importer) EnsureChannel ¶
EnsureChannel creates the channel with its historical creation time unless a channel with spec.ID already exists (then it is left as is). Name, slug and description go through ex's channel validators. Returns whether it was created; the stored channel is (re)indexed for search either way.
func (*Importer) EnsureConversation ¶
func (im *Importer) EnsureConversation(ctx context.Context, spec ConversationSpec) (string, bool, error)
EnsureConversation creates the conversation (already activated, so every participant sees it) unless it exists. Its ID is derived from the participants exactly as ex derives it, so an existing conversation between the same people is reused. Returns the ID and whether it was created.
func (*Importer) FinalizeParent ¶
func (im *Importer) FinalizeParent(ctx context.Context, spec FinalizeSpec) (FinalizeResult, error)
FinalizeParent brings a parent's derived state in line with imported messages. Thread roots and replies are read from the parent's partition with strongly consistent reads (one pass for all touched threads), so replies written just before are counted — the thread index (GSI1) can lag behind a bulk write and would leave threads with too low a reply count:
- each touched thread root gets its reply count, last reply time and recent reply authors recomputed from all its replies;
- the root author and every reply author follow the thread (an explicit unfollow is never overwritten): the reply path finds a thread's audience in its follow rows;
- the parent's unread counter and every member's read watermark advance by NewTopLevel together, so imported history is read and existing unread counts don't change;
- a conversation's activity time moves forward to LastActivityAt.
Calling it twice for the same import advances both counters twice, which leaves unread counts unchanged.
func (*Importer) GetMessages ¶
func (im *Importer) GetMessages(ctx context.Context, parentID string, ids []string) ([]Message, error)
GetMessages returns the given messages of one parent (batched reads). Missing IDs are absent from the result; order is not guaranteed.
func (*Importer) IndexParent ¶
IndexParent writes a parent's messages and the files they reference to the search index (tombstoned, system and no-index messages are skipped). No-op without search. Returns how many message and file documents were written.
func (*Importer) InvalidateUsers ¶
InvalidateUsers drops ex's cached records for the given users so the next read sees what was written. No-op without a cache.
func (*Importer) ParentStats ¶
func (im *Importer) ParentStats(ctx context.Context, parentID, parentType string) (ParentStats, error)
ParentStats counts what a parent holds. It walks the parent's whole message partition (one Query page per statsPageSize messages) plus its pin and file index rows and members' read state, so it costs roughly one read unit per 4 KB of messages: use it for validation, not on a hot path.
func (*Importer) PutAttachment ¶
func (im *Importer) PutAttachment(ctx context.Context, spec AttachmentSpec, open func(context.Context) (io.ReadCloser, error)) (bool, error)
PutAttachment stores a file and its attachment record. open is only called when the attachment does not exist yet, so re-runs skip the download. The body is spooled to a temporary file while its SHA-256 is computed, then uploaded in one checksummed PUT — the record is written already verified, so ex serves it without re-reading (image thumbnails are still generated lazily on first view). Returns whether it was created.
func (*Importer) PutMessages ¶
func (im *Importer) PutMessages(ctx context.Context, parentID, parentType string, specs []MessageSpec) (PutResult, error)
PutMessages inserts the messages that don't exist yet into a parent. Messages already present (by ID) are skipped untouched — use UpdateMessage or TombstoneMessage to change them. Every spec is validated before anything is written. The new messages' side rows (attachment references, the parent's file and pin index) are written first and the message rows (with the thread index) last, in batches, so a crash midway leaves nothing a re-run won't complete.
The message rows are written unconditionally after the existence check, so calls for the same parent must not run concurrently (see the package doc).
func (*Importer) RemoveChannel ¶
func (im *Importer) RemoveChannel(ctx context.Context, spec RemoveChannelSpec) (RemoveResult, error)
RemoveChannel deletes a channel the import created: all its messages (as RemoveMessages, without counter updates), memberships on both sides, its slug lock, everything else in its partition and its search document.
func (*Importer) RemoveConversation ¶
func (im *Importer) RemoveConversation(ctx context.Context, spec RemoveConversationSpec) (RemoveResult, error)
RemoveConversation deletes a conversation the import created: its messages (as RemoveMessages, without counter updates), its metadata and member rows, and every participant's sidebar row.
func (*Importer) RemoveMembers ¶
func (im *Importer) RemoveMembers(ctx context.Context, channelID string, userIDs []string) (err error)
RemoveMembers removes users from a channel (both membership rows). Users who aren't members are skipped. The channel's next message no longer alerts them.
func (*Importer) RemoveMessages ¶
func (im *Importer) RemoveMessages(ctx context.Context, parentID, parentType string, ids []string) (RemoveResult, error)
RemoveMessages hard-deletes messages of one parent together with their pin and file index rows, their references on attachments (an attachment no message references any more is deleted with its stored objects), thread follows of removed roots and their search documents. Thread roots that lost replies get their thread metadata recomputed, and the parent's unread counter and every member's read watermark go down by the number of removed top-level, non-system messages (never below zero, and no watermark above the counter), so unread counts stay as they were.
func (*Importer) RemoveUser ¶
func (im *Importer) RemoveUser(ctx context.Context, spec RemoveUserSpec) error
RemoveUser deletes a user the import created: profile, email index, search document and cached record. Memberships are not touched — remove them first (Force only skips the in-use check).
func (*Importer) TombstoneMessage ¶
TombstoneMessage soft-deletes a message the way ex does (the row stays so threads keep their shape; body, attachments, reactions and pin are cleared), drops its attachment references, file and pin index rows, and removes it from search. Already-deleted or unknown messages are a no-op.
func (*Importer) UpdateMessage ¶
func (im *Importer) UpdateMessage(ctx context.Context, parentID, msgID string, p MessagePatch) error
UpdateMessage applies a patch to an existing message, keeping the pin index in step. Returns store.ErrNotFound (wrapped) for an unknown message.
func (*Importer) UpsertEmoji ¶ added in v0.0.105
UpsertEmoji creates a workspace custom emoji from image unless one with spec.Name exists; an existing emoji is left untouched. The image is checked and kept the way ex keeps an upload: a still picture is shrunk to 256px and re-encoded as lossless WebP, an animated one (GIF, animated WebP) is copied. Messages and reactions refer to custom emoji by name, so history imported with ":name:" shows the emoji once it exists. Connected clients see it after their next reload (no event is published). Returns whether it was created.
func (*Importer) UpsertUser ¶
UpsertUser creates a single-sign-on member account (no password), or a bot account for spec.Bot, when no user with spec.ID exists yet. An existing user with that ID is left untouched, except that spec.Bot turns it into a bot account; an existing user with the same email but another ID yields *EmailTakenError. The display name is trimmed and capped like the sign-in flow does. A person is (re)indexed for search either way, a bot is removed from it.
type MemberSpec ¶
type MemberSpec struct {
UserID string
Role model.ChannelRole // 0 = member
JoinedAt time.Time
Muted bool
Favorite bool
}
MemberSpec describes one channel member.
type Message ¶
type Message struct {
ID string
AuthorID string
Body string
CreatedAt time.Time
EditedAt *time.Time
Deleted bool
System bool
ParentMessageID string
Reactions map[string][]string
Pinned bool
AttachmentIDs []string
ReplyCount int
}
Message is the current state of a stored message.
type MessagePatch ¶
type MessagePatch struct {
Body *string
EditedAt *time.Time
// Reactions, when non-nil, replaces the whole reaction map.
Reactions map[string][]string
Pinned *bool
PinnedBy string
PinnedAt *time.Time
}
MessagePatch lists the fields an incremental re-sync may change. Nil fields are left as they are.
type MessageSpec ¶
type MessageSpec struct {
ID string
AuthorID string
Body string
CreatedAt time.Time
EditedAt *time.Time
ParentMessageID string // thread root, for replies
AttachmentIDs []string
// Reactions maps an emoji key (unicode or ":shortcode:") to user IDs.
Reactions map[string][]string
Pinned bool
PinnedAt *time.Time
PinnedBy string
// System marks a system notice (AuthorID should be SystemAuthorID).
System bool
WebhookUsername string
WebhookAvatarURL string
WebhookIconEmoji string
MessageAttachments []model.MessageAttachment
}
MessageSpec describes one message. ID must be a ULID whose timestamp is CreatedAt (to the millisecond): ex orders messages by ID.
type ObjectStore ¶
type ObjectStore interface {
PutChecksummedObject(ctx context.Context, key, contentType string, body io.ReadSeeker, size int64, sha256Hex string) (string, error)
HeadObject(ctx context.Context, key string) (bool, error)
DeleteObject(ctx context.Context, key string) error
}
ObjectStore is the object storage holding attachment bodies: uploads, existence checks (validation) and deletes (rollback).
type Options ¶
type Options struct {
AWSRegion string
DynamoDBTable string
DynamoDBEndpoint string
S3Bucket string
S3Region string
S3Endpoint string
S3AccessKey string
S3SecretKey string
OpenSearchURL string
OpenSearchAWSRegion string // set to sign requests with SigV4 (AWS-managed domains)
RedisURL string
// TempDir holds attachment bodies while they are hashed before upload
// (empty = the OS temp dir).
TempDir string
}
Options are the connection settings Open uses. DynamoDBTable and S3Bucket are required; everything else is optional (empty endpoints mean AWS defaults, empty search/Redis settings disable those integrations).
type ParentStats ¶
type ParentStats struct {
TopLevel int // live, non-system top-level messages
Replies int // live thread replies
System int // live system messages
Deleted int // tombstoned messages
Pinned int // pin index rows
Files int // file index rows
Members int
// MessageSeq is the parent's unread counter; MembersUnread counts members
// whose read watermark is behind it.
MessageSeq int64
MembersUnread int
// ThreadsMismatched counts thread roots whose stored reply count differs
// from the number of their replies. A deleted reply stays in its thread
// as a placeholder and keeps counting, as it does in ex (a tombstoned
// root likewise keeps its count).
ThreadsMismatched int
}
ParentStats describes a channel or conversation as stored.
type PutResult ¶
type PutResult struct {
Inserted int
Existing int
// TopLevelInserted counts inserted top-level, non-system messages — the
// number FinalizeParent adds to the unread counters.
TopLevelInserted int
// ThreadRootIDs lists the roots that received inserted replies.
ThreadRootIDs []string
// LastCreatedAt is the newest inserted message's time (zero if none).
LastCreatedAt time.Time
}
PutResult summarizes a PutMessages call.
type RemoveChannelSpec ¶
RemoveChannelSpec names a channel to remove. ImportedMessageIDs are the messages the import wrote; any other message in the channel makes the call fail with ForeignMessagesError unless Force is set.
type RemoveConversationSpec ¶
RemoveConversationSpec names a conversation to remove; see RemoveChannelSpec.
type RemoveResult ¶
type RemoveResult struct {
Messages int // message rows deleted
Attachments int // attachments deleted (no message referenced them any more)
}
RemoveResult summarizes a removal.
type RemoveUserSpec ¶
RemoveUserSpec names a user to remove. Without Force the call refuses a user who still belongs to any channel or conversation, or who has signed in.
type SearchIndex ¶
type SearchIndex interface {
search.BulkWriter
DeleteDoc(ctx context.Context, index, id string) error
}
SearchIndex is the search integration: per-document upserts for users and channels, bulk writes for a parent's messages and files, and deletes for tombstoned messages.
type SlugTakenError ¶
SlugTakenError reports that another channel already holds the slug.
func (*SlugTakenError) Error ¶
func (e *SlugTakenError) Error() string
type User ¶
type User struct {
ID string `json:"id"`
Email string `json:"email"` // lowercased
DisplayName string `json:"displayName"`
Status string `json:"status"` // "active" | "deactivated"
SystemRole string `json:"systemRole"`
AuthProvider string `json:"authProvider"`
}
User is an existing ex user as seen by the importer.
type UserInUseError ¶
UserInUseError is returned when a user still has memberships or has signed in, and removal was not forced.
func (*UserInUseError) Error ¶
func (e *UserInUseError) Error() string
type UserSpec ¶
type UserSpec struct {
ID string
Email string
DisplayName string
Deactivated bool
CreatedAt time.Time
// Bot makes a machine account, like the ones ex creates for incoming
// webhooks: nobody signs in as it, and it is left out of search, the
// people directory and directory sync. Email is ignored: a bot gets an
// undeliverable ID@bots.ex.invalid address.
Bot bool
}
UserSpec describes a user to create.