lib

package
v0.10.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2026 License: MIT Imports: 41 Imported by: 0

Documentation

Index

Constants

View Source
const (
	MediaPodcast = "podcast"
	MediaVideos  = "videos"
)

Media kinds tracked in the manifest.

View Source
const DefaultBurst = 5

DefaultBurst defines the default burst size for the rate limiter.

View Source
const DefaultRatePerSecond = 2

DefaultRatePerSecond defines the default request rate per second when creating a new Fetcher.

View Source
const MediaManifestFile = ".sbstck-dl-media.json"

MediaManifestFile is the name of the media manifest in the output directory.

Variables

This section is empty.

Functions

func EmbedMedia added in v0.9.0

func EmbedMedia(postHTML, postVideoID string, media EmbeddedMedia) string

EmbedMedia embeds downloaded media in post HTML, as players of the local files. Native video placeholders are replaced with their videos. The post's own video, which usually has no placeholder, and the podcast audio are added at the start. Only the media elements are changed, so the rest of the HTML stays as it is.

func EmbedMediaInFile added in v0.9.0

func EmbedMediaInFile(path, postVideoID string, media EmbeddedMedia) error

EmbedMediaInFile adds local players to an existing HTML post through atomic publication. Missing posts are errors so callers keep output work retryable.

func ExportEPUB added in v0.10.0

func ExportEPUB(ctx context.Context, options EPUBOptions) error

ExportEPUB publishes one complete book, preserving the source archive and any previous book if preparing or publishing the replacement fails.

Types

type Archive added in v0.8.0

type Archive struct {
	Entries []ArchiveEntry
}

Archive represents a collection of posts for the archive page

func LoadArchive added in v0.8.0

func LoadArchive(dir, format string) (*Archive, error)

LoadArchive restores metadata and discovers legacy files without network requests. Unreadable or corrupt state is an error: never replace an index using an incomplete restoration.

func NewArchive added in v0.8.0

func NewArchive() *Archive

NewArchive creates a new Archive instance

func (*Archive) AddEntry added in v0.8.0

func (a *Archive) AddEntry(post Post, filePath string, downloadTime time.Time)

AddEntry adds a new entry to the archive, sorted by publication date (newest first)

func (*Archive) GenerateHTML added in v0.8.0

func (a *Archive) GenerateHTML(outputDir string) error

GenerateHTML creates an HTML archive page

func (*Archive) GenerateMarkdown added in v0.8.0

func (a *Archive) GenerateMarkdown(outputDir string) error

GenerateMarkdown creates a Markdown archive page

func (*Archive) GenerateText added in v0.8.0

func (a *Archive) GenerateText(outputDir string) error

GenerateText creates a plain text archive page

func (*Archive) LocalizeCovers added in v0.8.0

func (a *Archive) LocalizeCovers(ctx context.Context, downloader *ImageDownloader, bodyImages map[string]*ImageDownloadResult) ([]error, error)

LocalizeCovers also visits retained entries when no post bodies were fetched. Only verified successful results from this invocation may supply body assets. Ordinary cover failures are warnings; cancellation stops archive publication.

func (*Archive) RemoveEntry added in v0.8.0

func (a *Archive) RemoveEntry(path string)

RemoveEntry removes a local post identity from the index.

func (*Archive) SaveMetadata added in v0.8.0

func (a *Archive) SaveMetadata(dir, format string) error

SaveMetadata publishes portable, format-specific metadata for future runs.

type ArchiveEntry added in v0.8.0

type ArchiveEntry struct {
	Post            Post
	FilePath        string
	DownloadTime    time.Time
	LocalCoverImage string
	MediaUpdated    bool // Transient: body media belongs to a successfully written post this run.
}

ArchiveEntry represents a single entry in the archive page

func (ArchiveEntry) CoverReference added in v0.8.0

func (entry ArchiveEntry) CoverReference(outputDir string) string

LocalCoverImage is stored relative to the archive folder; CoverImage remains the remote source so missing assets can be downloaded again.

type ArchivePostWrite added in v0.8.0

type ArchivePostWrite struct {
	Format       string
	Path         string
	DownloadTime time.Time
	Err          error
}

ArchivePostWrite records the outcome of one format's post write.

type ArchivePublicationResult added in v0.8.0

type ArchivePublicationResult struct {
	Warnings         []error
	PublishedFormats []string
	Err              error
}

ArchivePublicationResult separates ordinary cover warnings from publication failures.

type ArchiveWorkflow added in v0.8.0

type ArchiveWorkflow struct {
	// contains filtered or unexported fields
}

ArchiveWorkflow owns incremental archive state for one download invocation. Post extraction, media preparation and post writes remain with the caller. The caller records completed post attempts serially, then publishes the archives.

func LoadArchiveWorkflow added in v0.8.0

func LoadArchiveWorkflow(dir string, formats []string) (*ArchiveWorkflow, error)

LoadArchiveWorkflow restores every selected format before any download work.

func (*ArchiveWorkflow) Publish added in v0.8.0

Publish shares cover work across formats, then publishes each independently. A metadata failure prevents that format's index from being replaced. A nil downloader leaves cover references unchanged. Otherwise, its output directory must match the directory used to load this workflow.

func (*ArchiveWorkflow) RecordPost added in v0.8.0

func (w *ArchiveWorkflow) RecordPost(post Post, images *ImageDownloadResult, writes []ArchivePostWrite)

RecordPost records one media preparation event and all its write outcomes. Even an event with no successful writes can invalidate a retained asset's proof. Writes describe attempted outputs in the selected formats; nil writes records media evidence alone when preparation prevented writing any output.

type AudioUpload added in v0.9.0

type AudioUpload struct {
	Id              string           `json:"id"`
	Duration        float64          `json:"duration"`
	IsFreePreview   bool             `json:"is_free_preview,omitempty"`
	FullPodcastInfo *FullPodcastInfo `json:"full_podcast_info,omitempty"`
}

type DateFilterFunc

type DateFilterFunc func(string) bool

type EPUBOptions added in v0.10.0

type EPUBOptions struct {
	Input    string
	Output   string
	Title    string
	Author   string
	Language string
}

EPUBOptions describes one offline export of a local HTML archive.

type EmbeddedMedia added in v0.9.0

type EmbeddedMedia struct {
	Podcast string            // podcast audio
	Videos  map[string]string // videos by upload ID
}

EmbeddedMedia lists downloaded media to embed in post HTML, as URLs relative to the post file.

type ExtractResult

type ExtractResult struct {
	Post Post
	Err  error
}

type Extractor

type Extractor struct {
	// contains filtered or unexported fields
}

Extractor is a utility for extracting Substack posts from URLs.

func NewExtractor

func NewExtractor(f *Fetcher) *Extractor

NewExtractor creates a new Extractor with the provided Fetcher. If the Fetcher is nil, a default Fetcher will be used.

func (*Extractor) ExtractAllPosts

func (e *Extractor) ExtractAllPosts(ctx context.Context, urls []string) <-chan ExtractResult

ExtractAllPosts extracts all posts from the given URLs using a worker pool pattern to limit concurrency and avoid overwhelming system resources.

func (*Extractor) ExtractPost

func (e *Extractor) ExtractPost(ctx context.Context, pageUrl string) (Post, error)

func (*Extractor) GetAllPostsURLs

func (e *Extractor) GetAllPostsURLs(ctx context.Context, pubURL string, f DateFilterFunc) ([]string, error)

GetAllPostsURLs discovers the publication's publicly enumerable archive. A feed or the initial archive HTML page cannot establish completeness.

type FetchError

type FetchError struct {
	TooManyRequests bool
	RetryAfter      int
	StatusCode      int
}

FetchError represents an error returned when encountering too many requests with a Retry-After value.

func (*FetchError) Error

func (e *FetchError) Error() string

Error returns the error message for the FetchError.

type FetchResult

type FetchResult struct {
	Url   string
	Body  io.ReadCloser
	Error error
}

FetchResult represents the result of a URL fetch operation.

type Fetcher

type Fetcher struct {
	Client      *http.Client
	RateLimiter *rate.Limiter
	BackoffCfg  backoff.BackOff
	Cookie      *http.Cookie
	MaxWorkers  int
	// contains filtered or unexported fields
}

Fetcher represents a URL fetcher with rate limiting and retry mechanisms.

func NewFetcher

func NewFetcher(opts ...FetcherOption) *Fetcher

NewFetcher creates a new Fetcher with the provided options.

func (*Fetcher) FetchURL

func (f *Fetcher) FetchURL(ctx context.Context, url string) (io.ReadCloser, error)

FetchURL fetches the specified URL with retries and rate limiting.

func (*Fetcher) FetchURLs

func (f *Fetcher) FetchURLs(ctx context.Context, urls []string) <-chan FetchResult

FetchURLs concurrently fetches the specified URLs and returns a channel to receive the FetchResults.

func (*Fetcher) ResolveRedirect added in v0.9.0

func (f *Fetcher) ResolveRedirect(ctx context.Context, url string) (string, error)

ResolveRedirect requests the URL without following redirects and returns the redirect target, or the URL itself if the response is not a redirect. Rate limiting (429), server errors and network errors are retried.

type FetcherOption

type FetcherOption func(*FetcherOptions)

FetcherOption defines a function that applies a specific option to FetcherOptions.

func WithBackOffConfig

func WithBackOffConfig(b backoff.BackOff) FetcherOption

WithBackOffConfig sets the backoff configuration for the Fetcher. Configure the policy before fetching. Exponential and constant policies are copied per request; retry loops using opaque custom policies are serialized.

func WithBurst added in v0.3.3

func WithBurst(burst int) FetcherOption

WithBurst sets the burst size for the rate limiter.

func WithCookie

func WithCookie(cookie *http.Cookie) FetcherOption

WithCookie sets the cookie for the Fetcher.

func WithMaxWorkers added in v0.3.3

func WithMaxWorkers(workers int) FetcherOption

WithMaxWorkers sets the maximum number of concurrent workers.

func WithProxyURL

func WithProxyURL(proxyURL *url.URL) FetcherOption

WithProxyURL sets the proxy URL for the Fetcher.

func WithRatePerSecond

func WithRatePerSecond(rate int) FetcherOption

WithRatePerSecond sets the rate per second for the Fetcher.

func WithTimeout added in v0.3.3

func WithTimeout(timeout time.Duration) FetcherOption

WithTimeout sets the HTTP client timeout.

type FetcherOptions

type FetcherOptions struct {
	RatePerSecond int
	Burst         int
	ProxyURL      *url.URL
	BackOffConfig backoff.BackOff
	Cookie        *http.Cookie
	Timeout       time.Duration
	MaxWorkers    int
}

FetcherOptions holds configurable options for Fetcher.

type FileDownloadResult added in v0.6.4

type FileDownloadResult struct {
	Files       []FileInfo
	UpdatedHTML string
	Success     int
	Failed      int
}

FileDownloadResult contains the results of downloading file attachments for a post

type FileDownloader added in v0.6.4

type FileDownloader struct {
	// contains filtered or unexported fields
}

FileDownloader handles downloading file attachments from Substack posts

func NewFileDownloader added in v0.6.4

func NewFileDownloader(fetcher *Fetcher, outputDir, filesDir string, extensions []string) *FileDownloader

NewFileDownloader creates a new FileDownloader instance

func (*FileDownloader) DownloadFiles added in v0.6.4

func (fd *FileDownloader) DownloadFiles(ctx context.Context, htmlContent string, postSlug string) (*FileDownloadResult, error)

DownloadFiles downloads all file attachments from a post's HTML content and returns updated HTML

type FileElement added in v0.6.4

type FileElement struct {
	DownloadURL string
	LocalPath   string
	Filename    string
	Success     bool
}

FileElement represents a file attachment element with its download URL and local path info

type FileInfo added in v0.6.4

type FileInfo struct {
	OriginalURL string
	LocalPath   string
	Filename    string
	Size        int64
	Success     bool
	Error       error
}

FileInfo represents information about a downloaded file attachment

type FullPodcastInfo added in v0.9.0

type FullPodcastInfo struct {
	Duration float64 `json:"duration"`
}

FullPodcastInfo describes the full episode when the served upload is a preview.

type ImageDownloadResult added in v0.5.0

type ImageDownloadResult struct {
	Images      []ImageInfo
	UpdatedHTML string
	Success     int
	Failed      int
}

ImageDownloadResult contains the results of downloading images for a post

type ImageDownloader added in v0.5.0

type ImageDownloader struct {
	// contains filtered or unexported fields
}

ImageDownloader handles downloading and processing images from Substack posts

func NewImageDownloader added in v0.5.0

func NewImageDownloader(fetcher *Fetcher, outputDir, imagesDir string, quality ImageQuality) *ImageDownloader

NewImageDownloader creates a new ImageDownloader instance

func (*ImageDownloader) DownloadImages added in v0.5.0

func (id *ImageDownloader) DownloadImages(ctx context.Context, htmlContent string, postSlug string) (*ImageDownloadResult, error)

DownloadImages downloads all images from a post's HTML content and returns updated HTML

type ImageElement added in v0.6.0

type ImageElement struct {
	BestURL   string   // The URL to download (highest quality)
	AllURLs   []string // All URLs that should be replaced with the local path
	LocalPath string   // Local path after download
	Success   bool     // Whether download was successful
}

ImageElement represents an image element with all its URLs

type ImageInfo added in v0.5.0

type ImageInfo struct {
	OriginalURL  string
	OriginalPath string // Downloaded source bytes, before any browser preview
	LocalPath    string
	Width        int
	Height       int
	Format       string
	Success      bool
	Error        error
}

ImageInfo contains information about a downloaded image

type ImageQuality added in v0.5.0

type ImageQuality string

ImageQuality represents the quality level for image downloads

const (
	ImageQualityHigh   ImageQuality = "high"   // 1456w
	ImageQualityMedium ImageQuality = "medium" // 848w
	ImageQualityLow    ImageQuality = "low"    // 424w
)

type MediaManifest added in v0.9.0

type MediaManifest struct {
	Version int                     `json:"version"`
	Posts   map[string]*MediaRecord `json:"posts"` // keyed by post slug
	// contains filtered or unexported fields
}

MediaManifest records which posts' media downloads are complete, so that later runs don't need to fetch those posts again. It lives in the output directory; deleting it makes the next run check every post's media again.

func LoadMediaManifest added in v0.9.0

func LoadMediaManifest(outputDir string) (*MediaManifest, error)

LoadMediaManifest loads the manifest from outputDir, or returns an empty one if there is none yet.

func NewMediaManifest added in v0.9.0

func NewMediaManifest(outputDir string) *MediaManifest

NewMediaManifest returns an empty manifest for outputDir.

func (*MediaManifest) IsComplete added in v0.9.0

func (m *MediaManifest) IsComplete(slug, kind string) bool

IsComplete reports whether the given kind of media is complete for the post: recorded as complete, with all its files still present.

func (*MediaManifest) Record added in v0.9.0

func (m *MediaManifest) Record(slug, kind string, complete bool, files []string)

Record stores the status of a kind of media for the post. Local paths in files are stored relative to the output directory.

func (*MediaManifest) Save added in v0.9.0

func (m *MediaManifest) Save() error

Save writes the manifest to the output directory, atomically.

type MediaRecord added in v0.9.0

type MediaRecord map[string]*MediaStatus

MediaRecord holds the media status of one post, by media kind.

type MediaStatus added in v0.9.0

type MediaStatus struct {
	// Complete is true after successful output publication, when all media was
	// downloaded in full, or the post has none. Previews, unavailable or
	// failed downloads leave it false so the post is checked again.
	Complete bool `json:"complete"`
	// Files lists the downloaded files, relative to the output directory.
	Files []string `json:"files,omitempty"`
}

MediaStatus is the outcome of downloading one kind of media for a post.

type PodcastDownloadResult added in v0.9.0

type PodcastDownloadResult struct {
	LocalPath string
	Size      int64
	Preview   bool // true if only a truncated preview was available
	Existed   bool // true if the file was already present and not re-downloaded
}

PodcastDownloadResult describes the outcome of downloading a post's podcast audio.

func (*PodcastDownloadResult) Source added in v0.9.0

func (r *PodcastDownloadResult) Source(dir string) string

Source returns the path of the downloaded audio relative to dir, as a URL for HTML files in dir to reference.

type PodcastDownloader added in v0.9.0

type PodcastDownloader struct {
	// contains filtered or unexported fields
}

PodcastDownloader downloads podcast audio from Substack posts.

func NewPodcastDownloader added in v0.9.0

func NewPodcastDownloader(fetcher *Fetcher, outputDir, podcastsDir string) *PodcastDownloader

NewPodcastDownloader creates a new PodcastDownloader instance.

func (*PodcastDownloader) DownloadPodcast added in v0.9.0

func (pd *PodcastDownloader) DownloadPodcast(ctx context.Context, post Post, baseName string) (*PodcastDownloadResult, error)

DownloadPodcast downloads the post's podcast audio to <outputDir>/<podcastsDir>/<baseName>.mp3 (or another audio extension, depending on the format served). Truncated previews are saved as <baseName>_preview.mp3 so that a later run with a subscriber cookie still fetches the full episode. Returns nil and no error if the post has no podcast.

type PodcastUpload added in v0.9.0

type PodcastUpload = AudioUpload

PodcastUpload holds the subset of Substack's podcastUpload metadata needed to tell a full episode apart from a free preview.

type Post

type Post struct {
	Id               int          `json:"id"`
	PublicationId    int          `json:"publication_id"`
	Type             string       `json:"type"`
	Slug             string       `json:"slug"`
	PostDate         string       `json:"post_date"`
	CanonicalUrl     string       `json:"canonical_url"`
	PreviousPostSlug string       `json:"previous_post_slug"`
	NextPostSlug     string       `json:"next_post_slug"`
	CoverImage       string       `json:"cover_image"`
	Description      string       `json:"description"`
	Subtitle         string       `json:"subtitle,omitempty"`
	WordCount        int          `json:"wordcount"`
	Title            string       `json:"title"`
	BodyHTML         string       `json:"body_html"`
	PodcastUrl       string       `json:"podcast_url,omitempty"`
	VideoUpload      *VideoUpload `json:"videoUpload,omitempty"`
	PodcastUpload    *AudioUpload `json:"podcastUpload,omitempty"`
	VoiceoverUpload  *AudioUpload `json:"voiceoverUpload,omitempty"`
	Audience         string       `json:"audience,omitempty"`
	PodcastDuration  float64      `json:"podcast_duration,omitempty"`
	VideoUploadId    string       `json:"video_upload_id,omitempty"`
}

Post represents a structured Substack post with various fields.

func (*Post) HasPodcast added in v0.9.0

func (p *Post) HasPodcast() bool

HasPodcast reports whether the post has podcast audio attached.

func (*Post) IsPaywalled added in v0.9.0

func (p *Post) IsPaywalled() bool

IsPaywalled reports whether the post is restricted to some subscribers. Without access, Substack may withhold its media without any sign of it.

func (*Post) IsPodcastPreview added in v0.9.0

func (p *Post) IsPodcastPreview() bool

IsPodcastPreview reports whether the served audio is a truncated preview of a paid episode. Substack marks some full-length episodes as previews too (e.g. ad-supported free versions), so we only treat it as a preview when it is actually shorter than the full episode.

func (*Post) IsPodcastUnavailable added in v0.9.0

func (p *Post) IsPodcastUnavailable() bool

IsPodcastUnavailable reports whether the post is a podcast episode whose audio Substack withheld. Some publications serve no preview of paid episodes at all to readers without access: the post then has no podcast_url.

func (Post) PrepareMedia added in v0.8.0

func (p Post) PrepareMedia(ctx context.Context, outputDir string,
	downloadImages bool, imageQuality ImageQuality, imagesDir string,
	downloadFiles bool, fileExtensions []string, filesDir string, fetcher *Fetcher) (Post, *ImageDownloadResult, error)

PrepareMedia downloads and rewrites a canonical HTML body once for all outputs. Plain text continues to render the original body, without local image links.

func (*Post) ToHTML

func (p *Post) ToHTML(withTitle bool) string

ToHTML returns the Post's HTML body as-is or with an optional title header.

func (*Post) ToJSON

func (p *Post) ToJSON() (string, error)

ToJSON converts the Post to a JSON string.

func (*Post) ToMD

func (p *Post) ToMD(withTitle bool) (string, error)

ToMD converts the Post's HTML body to Markdown format.

func (*Post) ToText

func (p *Post) ToText(withTitle bool) string

ToText converts the Post's HTML body to plain text format.

func (*Post) WriteToFile

func (p *Post) WriteToFile(path string, format string, addSourceURL bool) error

WriteToFile writes the Post's content to a file in the specified format (html, md, or txt).

func (*Post) WriteToFileWithImages added in v0.5.0

func (p *Post) WriteToFileWithImages(ctx context.Context, path string, format string, addSourceURL bool,
	downloadImages bool, imageQuality ImageQuality, imagesDir string,
	downloadFiles bool, fileExtensions []string, filesDir string, fetcher *Fetcher) (*ImageDownloadResult, error)

WriteToFileWithImages retains the single-output API over the shared media pass.

type PostWrapper

type PostWrapper struct {
	Post Post `json:"post"`
}

PostWrapper wraps a Post object for JSON unmarshaling.

type RawPost

type RawPost struct {
	// contains filtered or unexported fields
}

RawPost represents a raw Substack post in string format.

func (*RawPost) ToPost

func (r *RawPost) ToPost() (Post, error)

ToPost converts the RawPost to a structured Post object.

type VideoDownloadResult added in v0.9.0

type VideoDownloadResult struct {
	Videos  []VideoInfo
	Success int
	Failed  int
}

VideoDownloadResult contains the results of downloading the videos of a post.

func (*VideoDownloadResult) Sources added in v0.9.0

func (r *VideoDownloadResult) Sources(dir string) map[string]string

Sources maps the upload IDs of the downloaded videos to their file paths relative to dir, as URLs for HTML files in dir to reference.

type VideoDownloader added in v0.9.0

type VideoDownloader struct {
	// contains filtered or unexported fields
}

VideoDownloader downloads Substack-hosted videos: the video of a video post and native videos embedded in the post body. Third-party embeds such as YouTube are not downloaded.

func NewVideoDownloader added in v0.9.0

func NewVideoDownloader(fetcher *Fetcher, outputDir, videosDir string) *VideoDownloader

NewVideoDownloader creates a new VideoDownloader instance.

func (*VideoDownloader) DownloadVideos added in v0.9.0

func (vd *VideoDownloader) DownloadVideos(ctx context.Context, post Post, baseName string) (*VideoDownloadResult, error)

DownloadVideos downloads the post's videos to <outputDir>/<videosDir>/ as MP4 files named after baseName. Free previews get a _preview suffix so that a later run with a subscriber cookie still fetches the full video.

type VideoInfo added in v0.9.0

type VideoInfo struct {
	UploadID  string
	LocalPath string
	Size      int64
	Preview   bool // true if only a free preview was available
	Existed   bool // true if the file was already present and not re-downloaded
	Success   bool
	Error     error
}

VideoInfo describes the outcome of downloading a single video.

type VideoUpload added in v0.9.0

type VideoUpload struct {
	Id                   string       `json:"id"`
	Duration             float64      `json:"duration"`
	ExtractedAudio       *AudioUpload `json:"extractedAudio,omitempty"`
	MuxPreviewPlaybackID string       `json:"mux_preview_playback_id,omitempty"`
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
⌂ 换网址