Documentation
¶
Index ¶
- Constants
- func EmbedMedia(postHTML, postVideoID string, media EmbeddedMedia) string
- func EmbedMediaInFile(path, postVideoID string, media EmbeddedMedia) error
- func ExportEPUB(ctx context.Context, options EPUBOptions) error
- type Archive
- func (a *Archive) AddEntry(post Post, filePath string, downloadTime time.Time)
- func (a *Archive) GenerateHTML(outputDir string) error
- func (a *Archive) GenerateMarkdown(outputDir string) error
- func (a *Archive) GenerateText(outputDir string) error
- func (a *Archive) LocalizeCovers(ctx context.Context, downloader *ImageDownloader, ...) ([]error, error)
- func (a *Archive) RemoveEntry(path string)
- func (a *Archive) SaveMetadata(dir, format string) error
- type ArchiveEntry
- type ArchivePostWrite
- type ArchivePublicationResult
- type ArchiveWorkflow
- type AudioUpload
- type DateFilterFunc
- type EPUBOptions
- type EmbeddedMedia
- type ExtractResult
- type Extractor
- type FetchError
- type FetchResult
- type Fetcher
- type FetcherOption
- func WithBackOffConfig(b backoff.BackOff) FetcherOption
- func WithBurst(burst int) FetcherOption
- func WithCookie(cookie *http.Cookie) FetcherOption
- func WithMaxWorkers(workers int) FetcherOption
- func WithProxyURL(proxyURL *url.URL) FetcherOption
- func WithRatePerSecond(rate int) FetcherOption
- func WithTimeout(timeout time.Duration) FetcherOption
- type FetcherOptions
- type FileDownloadResult
- type FileDownloader
- type FileElement
- type FileInfo
- type FullPodcastInfo
- type ImageDownloadResult
- type ImageDownloader
- type ImageElement
- type ImageInfo
- type ImageQuality
- type MediaManifest
- type MediaRecord
- type MediaStatus
- type PodcastDownloadResult
- type PodcastDownloader
- type PodcastUpload
- type Post
- func (p *Post) HasPodcast() bool
- func (p *Post) IsPaywalled() bool
- func (p *Post) IsPodcastPreview() bool
- func (p *Post) IsPodcastUnavailable() bool
- func (p Post) PrepareMedia(ctx context.Context, outputDir string, downloadImages bool, ...) (Post, *ImageDownloadResult, error)
- func (p *Post) ToHTML(withTitle bool) string
- func (p *Post) ToJSON() (string, error)
- func (p *Post) ToMD(withTitle bool) (string, error)
- func (p *Post) ToText(withTitle bool) string
- func (p *Post) WriteToFile(path string, format string, addSourceURL bool) error
- func (p *Post) WriteToFileWithImages(ctx context.Context, path string, format string, addSourceURL bool, ...) (*ImageDownloadResult, error)
- type PostWrapper
- type RawPost
- type VideoDownloadResult
- type VideoDownloader
- type VideoInfo
- type VideoUpload
Constants ¶
const ( MediaPodcast = "podcast" MediaVideos = "videos" )
Media kinds tracked in the manifest.
const DefaultBurst = 5
DefaultBurst defines the default burst size for the rate limiter.
const DefaultRatePerSecond = 2
DefaultRatePerSecond defines the default request rate per second when creating a new Fetcher.
const MediaManifestFile = ".sbstck-dl-media.json"
MediaManifestFile is the name of the media manifest in the output directory.
Variables ¶
This section is empty.
Functions ¶
func EmbedMedia ¶ added in v0.9.0
func EmbedMedia(postHTML, postVideoID string, media EmbeddedMedia) string
EmbedMedia embeds downloaded media in post HTML, as players of the local files. Native video placeholders are replaced with their videos. The post's own video, which usually has no placeholder, and the podcast audio are added at the start. Only the media elements are changed, so the rest of the HTML stays as it is.
func EmbedMediaInFile ¶ added in v0.9.0
func EmbedMediaInFile(path, postVideoID string, media EmbeddedMedia) error
EmbedMediaInFile adds local players to an existing HTML post through atomic publication. Missing posts are errors so callers keep output work retryable.
func ExportEPUB ¶ added in v0.10.0
func ExportEPUB(ctx context.Context, options EPUBOptions) error
ExportEPUB publishes one complete book, preserving the source archive and any previous book if preparing or publishing the replacement fails.
Types ¶
type Archive ¶ added in v0.8.0
type Archive struct {
Entries []ArchiveEntry
}
Archive represents a collection of posts for the archive page
func LoadArchive ¶ added in v0.8.0
LoadArchive restores metadata and discovers legacy files without network requests. Unreadable or corrupt state is an error: never replace an index using an incomplete restoration.
func NewArchive ¶ added in v0.8.0
func NewArchive() *Archive
NewArchive creates a new Archive instance
func (*Archive) AddEntry ¶ added in v0.8.0
AddEntry adds a new entry to the archive, sorted by publication date (newest first)
func (*Archive) GenerateHTML ¶ added in v0.8.0
GenerateHTML creates an HTML archive page
func (*Archive) GenerateMarkdown ¶ added in v0.8.0
GenerateMarkdown creates a Markdown archive page
func (*Archive) GenerateText ¶ added in v0.8.0
GenerateText creates a plain text archive page
func (*Archive) LocalizeCovers ¶ added in v0.8.0
func (a *Archive) LocalizeCovers(ctx context.Context, downloader *ImageDownloader, bodyImages map[string]*ImageDownloadResult) ([]error, error)
LocalizeCovers also visits retained entries when no post bodies were fetched. Only verified successful results from this invocation may supply body assets. Ordinary cover failures are warnings; cancellation stops archive publication.
func (*Archive) RemoveEntry ¶ added in v0.8.0
RemoveEntry removes a local post identity from the index.
func (*Archive) SaveMetadata ¶ added in v0.8.0
SaveMetadata publishes portable, format-specific metadata for future runs.
type ArchiveEntry ¶ added in v0.8.0
type ArchiveEntry struct {
Post Post
FilePath string
DownloadTime time.Time
LocalCoverImage string
MediaUpdated bool // Transient: body media belongs to a successfully written post this run.
}
ArchiveEntry represents a single entry in the archive page
func (ArchiveEntry) CoverReference ¶ added in v0.8.0
func (entry ArchiveEntry) CoverReference(outputDir string) string
LocalCoverImage is stored relative to the archive folder; CoverImage remains the remote source so missing assets can be downloaded again.
type ArchivePostWrite ¶ added in v0.8.0
ArchivePostWrite records the outcome of one format's post write.
type ArchivePublicationResult ¶ added in v0.8.0
ArchivePublicationResult separates ordinary cover warnings from publication failures.
type ArchiveWorkflow ¶ added in v0.8.0
type ArchiveWorkflow struct {
// contains filtered or unexported fields
}
ArchiveWorkflow owns incremental archive state for one download invocation. Post extraction, media preparation and post writes remain with the caller. The caller records completed post attempts serially, then publishes the archives.
func LoadArchiveWorkflow ¶ added in v0.8.0
func LoadArchiveWorkflow(dir string, formats []string) (*ArchiveWorkflow, error)
LoadArchiveWorkflow restores every selected format before any download work.
func (*ArchiveWorkflow) Publish ¶ added in v0.8.0
func (w *ArchiveWorkflow) Publish(ctx context.Context, downloader *ImageDownloader) ArchivePublicationResult
Publish shares cover work across formats, then publishes each independently. A metadata failure prevents that format's index from being replaced. A nil downloader leaves cover references unchanged. Otherwise, its output directory must match the directory used to load this workflow.
func (*ArchiveWorkflow) RecordPost ¶ added in v0.8.0
func (w *ArchiveWorkflow) RecordPost(post Post, images *ImageDownloadResult, writes []ArchivePostWrite)
RecordPost records one media preparation event and all its write outcomes. Even an event with no successful writes can invalidate a retained asset's proof. Writes describe attempted outputs in the selected formats; nil writes records media evidence alone when preparation prevented writing any output.
type AudioUpload ¶ added in v0.9.0
type AudioUpload struct {
Id string `json:"id"`
Duration float64 `json:"duration"`
IsFreePreview bool `json:"is_free_preview,omitempty"`
FullPodcastInfo *FullPodcastInfo `json:"full_podcast_info,omitempty"`
}
type DateFilterFunc ¶
type EPUBOptions ¶ added in v0.10.0
EPUBOptions describes one offline export of a local HTML archive.
type EmbeddedMedia ¶ added in v0.9.0
type EmbeddedMedia struct {
Podcast string // podcast audio
Videos map[string]string // videos by upload ID
}
EmbeddedMedia lists downloaded media to embed in post HTML, as URLs relative to the post file.
type ExtractResult ¶
type Extractor ¶
type Extractor struct {
// contains filtered or unexported fields
}
Extractor is a utility for extracting Substack posts from URLs.
func NewExtractor ¶
NewExtractor creates a new Extractor with the provided Fetcher. If the Fetcher is nil, a default Fetcher will be used.
func (*Extractor) ExtractAllPosts ¶
func (e *Extractor) ExtractAllPosts(ctx context.Context, urls []string) <-chan ExtractResult
ExtractAllPosts extracts all posts from the given URLs using a worker pool pattern to limit concurrency and avoid overwhelming system resources.
func (*Extractor) ExtractPost ¶
func (*Extractor) GetAllPostsURLs ¶
func (e *Extractor) GetAllPostsURLs(ctx context.Context, pubURL string, f DateFilterFunc) ([]string, error)
GetAllPostsURLs discovers the publication's publicly enumerable archive. A feed or the initial archive HTML page cannot establish completeness.
type FetchError ¶
FetchError represents an error returned when encountering too many requests with a Retry-After value.
func (*FetchError) Error ¶
func (e *FetchError) Error() string
Error returns the error message for the FetchError.
type FetchResult ¶
type FetchResult struct {
Url string
Body io.ReadCloser
Error error
}
FetchResult represents the result of a URL fetch operation.
type Fetcher ¶
type Fetcher struct {
Client *http.Client
RateLimiter *rate.Limiter
BackoffCfg backoff.BackOff
Cookie *http.Cookie
MaxWorkers int
// contains filtered or unexported fields
}
Fetcher represents a URL fetcher with rate limiting and retry mechanisms.
func NewFetcher ¶
func NewFetcher(opts ...FetcherOption) *Fetcher
NewFetcher creates a new Fetcher with the provided options.
func (*Fetcher) FetchURLs ¶
func (f *Fetcher) FetchURLs(ctx context.Context, urls []string) <-chan FetchResult
FetchURLs concurrently fetches the specified URLs and returns a channel to receive the FetchResults.
func (*Fetcher) ResolveRedirect ¶ added in v0.9.0
ResolveRedirect requests the URL without following redirects and returns the redirect target, or the URL itself if the response is not a redirect. Rate limiting (429), server errors and network errors are retried.
type FetcherOption ¶
type FetcherOption func(*FetcherOptions)
FetcherOption defines a function that applies a specific option to FetcherOptions.
func WithBackOffConfig ¶
func WithBackOffConfig(b backoff.BackOff) FetcherOption
WithBackOffConfig sets the backoff configuration for the Fetcher. Configure the policy before fetching. Exponential and constant policies are copied per request; retry loops using opaque custom policies are serialized.
func WithBurst ¶ added in v0.3.3
func WithBurst(burst int) FetcherOption
WithBurst sets the burst size for the rate limiter.
func WithCookie ¶
func WithCookie(cookie *http.Cookie) FetcherOption
WithCookie sets the cookie for the Fetcher.
func WithMaxWorkers ¶ added in v0.3.3
func WithMaxWorkers(workers int) FetcherOption
WithMaxWorkers sets the maximum number of concurrent workers.
func WithProxyURL ¶
func WithProxyURL(proxyURL *url.URL) FetcherOption
WithProxyURL sets the proxy URL for the Fetcher.
func WithRatePerSecond ¶
func WithRatePerSecond(rate int) FetcherOption
WithRatePerSecond sets the rate per second for the Fetcher.
func WithTimeout ¶ added in v0.3.3
func WithTimeout(timeout time.Duration) FetcherOption
WithTimeout sets the HTTP client timeout.
type FetcherOptions ¶
type FetcherOptions struct {
RatePerSecond int
Burst int
ProxyURL *url.URL
BackOffConfig backoff.BackOff
Cookie *http.Cookie
Timeout time.Duration
MaxWorkers int
}
FetcherOptions holds configurable options for Fetcher.
type FileDownloadResult ¶ added in v0.6.4
FileDownloadResult contains the results of downloading file attachments for a post
type FileDownloader ¶ added in v0.6.4
type FileDownloader struct {
// contains filtered or unexported fields
}
FileDownloader handles downloading file attachments from Substack posts
func NewFileDownloader ¶ added in v0.6.4
func NewFileDownloader(fetcher *Fetcher, outputDir, filesDir string, extensions []string) *FileDownloader
NewFileDownloader creates a new FileDownloader instance
func (*FileDownloader) DownloadFiles ¶ added in v0.6.4
func (fd *FileDownloader) DownloadFiles(ctx context.Context, htmlContent string, postSlug string) (*FileDownloadResult, error)
DownloadFiles downloads all file attachments from a post's HTML content and returns updated HTML
type FileElement ¶ added in v0.6.4
FileElement represents a file attachment element with its download URL and local path info
type FileInfo ¶ added in v0.6.4
type FileInfo struct {
OriginalURL string
LocalPath string
Filename string
Size int64
Success bool
Error error
}
FileInfo represents information about a downloaded file attachment
type FullPodcastInfo ¶ added in v0.9.0
type FullPodcastInfo struct {
Duration float64 `json:"duration"`
}
FullPodcastInfo describes the full episode when the served upload is a preview.
type ImageDownloadResult ¶ added in v0.5.0
ImageDownloadResult contains the results of downloading images for a post
type ImageDownloader ¶ added in v0.5.0
type ImageDownloader struct {
// contains filtered or unexported fields
}
ImageDownloader handles downloading and processing images from Substack posts
func NewImageDownloader ¶ added in v0.5.0
func NewImageDownloader(fetcher *Fetcher, outputDir, imagesDir string, quality ImageQuality) *ImageDownloader
NewImageDownloader creates a new ImageDownloader instance
func (*ImageDownloader) DownloadImages ¶ added in v0.5.0
func (id *ImageDownloader) DownloadImages(ctx context.Context, htmlContent string, postSlug string) (*ImageDownloadResult, error)
DownloadImages downloads all images from a post's HTML content and returns updated HTML
type ImageElement ¶ added in v0.6.0
type ImageElement struct {
BestURL string // The URL to download (highest quality)
AllURLs []string // All URLs that should be replaced with the local path
LocalPath string // Local path after download
Success bool // Whether download was successful
}
ImageElement represents an image element with all its URLs
type ImageInfo ¶ added in v0.5.0
type ImageInfo struct {
OriginalURL string
OriginalPath string // Downloaded source bytes, before any browser preview
LocalPath string
Width int
Height int
Format string
Success bool
Error error
}
ImageInfo contains information about a downloaded image
type ImageQuality ¶ added in v0.5.0
type ImageQuality string
ImageQuality represents the quality level for image downloads
const ( ImageQualityHigh ImageQuality = "high" // 1456w ImageQualityMedium ImageQuality = "medium" // 848w ImageQualityLow ImageQuality = "low" // 424w )
type MediaManifest ¶ added in v0.9.0
type MediaManifest struct {
Version int `json:"version"`
Posts map[string]*MediaRecord `json:"posts"` // keyed by post slug
// contains filtered or unexported fields
}
MediaManifest records which posts' media downloads are complete, so that later runs don't need to fetch those posts again. It lives in the output directory; deleting it makes the next run check every post's media again.
func LoadMediaManifest ¶ added in v0.9.0
func LoadMediaManifest(outputDir string) (*MediaManifest, error)
LoadMediaManifest loads the manifest from outputDir, or returns an empty one if there is none yet.
func NewMediaManifest ¶ added in v0.9.0
func NewMediaManifest(outputDir string) *MediaManifest
NewMediaManifest returns an empty manifest for outputDir.
func (*MediaManifest) IsComplete ¶ added in v0.9.0
func (m *MediaManifest) IsComplete(slug, kind string) bool
IsComplete reports whether the given kind of media is complete for the post: recorded as complete, with all its files still present.
func (*MediaManifest) Record ¶ added in v0.9.0
func (m *MediaManifest) Record(slug, kind string, complete bool, files []string)
Record stores the status of a kind of media for the post. Local paths in files are stored relative to the output directory.
func (*MediaManifest) Save ¶ added in v0.9.0
func (m *MediaManifest) Save() error
Save writes the manifest to the output directory, atomically.
type MediaRecord ¶ added in v0.9.0
type MediaRecord map[string]*MediaStatus
MediaRecord holds the media status of one post, by media kind.
type MediaStatus ¶ added in v0.9.0
type MediaStatus struct {
// Complete is true after successful output publication, when all media was
// downloaded in full, or the post has none. Previews, unavailable or
// failed downloads leave it false so the post is checked again.
Complete bool `json:"complete"`
// Files lists the downloaded files, relative to the output directory.
Files []string `json:"files,omitempty"`
}
MediaStatus is the outcome of downloading one kind of media for a post.
type PodcastDownloadResult ¶ added in v0.9.0
type PodcastDownloadResult struct {
LocalPath string
Size int64
Preview bool // true if only a truncated preview was available
Existed bool // true if the file was already present and not re-downloaded
}
PodcastDownloadResult describes the outcome of downloading a post's podcast audio.
func (*PodcastDownloadResult) Source ¶ added in v0.9.0
func (r *PodcastDownloadResult) Source(dir string) string
Source returns the path of the downloaded audio relative to dir, as a URL for HTML files in dir to reference.
type PodcastDownloader ¶ added in v0.9.0
type PodcastDownloader struct {
// contains filtered or unexported fields
}
PodcastDownloader downloads podcast audio from Substack posts.
func NewPodcastDownloader ¶ added in v0.9.0
func NewPodcastDownloader(fetcher *Fetcher, outputDir, podcastsDir string) *PodcastDownloader
NewPodcastDownloader creates a new PodcastDownloader instance.
func (*PodcastDownloader) DownloadPodcast ¶ added in v0.9.0
func (pd *PodcastDownloader) DownloadPodcast(ctx context.Context, post Post, baseName string) (*PodcastDownloadResult, error)
DownloadPodcast downloads the post's podcast audio to <outputDir>/<podcastsDir>/<baseName>.mp3 (or another audio extension, depending on the format served). Truncated previews are saved as <baseName>_preview.mp3 so that a later run with a subscriber cookie still fetches the full episode. Returns nil and no error if the post has no podcast.
type PodcastUpload ¶ added in v0.9.0
type PodcastUpload = AudioUpload
PodcastUpload holds the subset of Substack's podcastUpload metadata needed to tell a full episode apart from a free preview.
type Post ¶
type Post struct {
Id int `json:"id"`
PublicationId int `json:"publication_id"`
Type string `json:"type"`
Slug string `json:"slug"`
PostDate string `json:"post_date"`
CanonicalUrl string `json:"canonical_url"`
PreviousPostSlug string `json:"previous_post_slug"`
NextPostSlug string `json:"next_post_slug"`
CoverImage string `json:"cover_image"`
Description string `json:"description"`
Subtitle string `json:"subtitle,omitempty"`
WordCount int `json:"wordcount"`
Title string `json:"title"`
BodyHTML string `json:"body_html"`
PodcastUrl string `json:"podcast_url,omitempty"`
VideoUpload *VideoUpload `json:"videoUpload,omitempty"`
PodcastUpload *AudioUpload `json:"podcastUpload,omitempty"`
VoiceoverUpload *AudioUpload `json:"voiceoverUpload,omitempty"`
Audience string `json:"audience,omitempty"`
PodcastDuration float64 `json:"podcast_duration,omitempty"`
VideoUploadId string `json:"video_upload_id,omitempty"`
}
Post represents a structured Substack post with various fields.
func (*Post) HasPodcast ¶ added in v0.9.0
HasPodcast reports whether the post has podcast audio attached.
func (*Post) IsPaywalled ¶ added in v0.9.0
IsPaywalled reports whether the post is restricted to some subscribers. Without access, Substack may withhold its media without any sign of it.
func (*Post) IsPodcastPreview ¶ added in v0.9.0
IsPodcastPreview reports whether the served audio is a truncated preview of a paid episode. Substack marks some full-length episodes as previews too (e.g. ad-supported free versions), so we only treat it as a preview when it is actually shorter than the full episode.
func (*Post) IsPodcastUnavailable ¶ added in v0.9.0
IsPodcastUnavailable reports whether the post is a podcast episode whose audio Substack withheld. Some publications serve no preview of paid episodes at all to readers without access: the post then has no podcast_url.
func (Post) PrepareMedia ¶ added in v0.8.0
func (p Post) PrepareMedia(ctx context.Context, outputDir string, downloadImages bool, imageQuality ImageQuality, imagesDir string, downloadFiles bool, fileExtensions []string, filesDir string, fetcher *Fetcher) (Post, *ImageDownloadResult, error)
PrepareMedia downloads and rewrites a canonical HTML body once for all outputs. Plain text continues to render the original body, without local image links.
func (*Post) WriteToFile ¶
WriteToFile writes the Post's content to a file in the specified format (html, md, or txt).
func (*Post) WriteToFileWithImages ¶ added in v0.5.0
func (p *Post) WriteToFileWithImages(ctx context.Context, path string, format string, addSourceURL bool, downloadImages bool, imageQuality ImageQuality, imagesDir string, downloadFiles bool, fileExtensions []string, filesDir string, fetcher *Fetcher) (*ImageDownloadResult, error)
WriteToFileWithImages retains the single-output API over the shared media pass.
type PostWrapper ¶
type PostWrapper struct {
Post Post `json:"post"`
}
PostWrapper wraps a Post object for JSON unmarshaling.
type RawPost ¶
type RawPost struct {
// contains filtered or unexported fields
}
RawPost represents a raw Substack post in string format.
type VideoDownloadResult ¶ added in v0.9.0
VideoDownloadResult contains the results of downloading the videos of a post.
type VideoDownloader ¶ added in v0.9.0
type VideoDownloader struct {
// contains filtered or unexported fields
}
VideoDownloader downloads Substack-hosted videos: the video of a video post and native videos embedded in the post body. Third-party embeds such as YouTube are not downloaded.
func NewVideoDownloader ¶ added in v0.9.0
func NewVideoDownloader(fetcher *Fetcher, outputDir, videosDir string) *VideoDownloader
NewVideoDownloader creates a new VideoDownloader instance.
func (*VideoDownloader) DownloadVideos ¶ added in v0.9.0
func (vd *VideoDownloader) DownloadVideos(ctx context.Context, post Post, baseName string) (*VideoDownloadResult, error)
DownloadVideos downloads the post's videos to <outputDir>/<videosDir>/ as MP4 files named after baseName. Free previews get a _preview suffix so that a later run with a subscriber cookie still fetches the full video.
type VideoInfo ¶ added in v0.9.0
type VideoInfo struct {
UploadID string
LocalPath string
Size int64
Preview bool // true if only a free preview was available
Existed bool // true if the file was already present and not re-downloaded
Success bool
Error error
}
VideoInfo describes the outcome of downloading a single video.
type VideoUpload ¶ added in v0.9.0
type VideoUpload struct {
Id string `json:"id"`
Duration float64 `json:"duration"`
ExtractedAudio *AudioUpload `json:"extractedAudio,omitempty"`
MuxPreviewPlaybackID string `json:"mux_preview_playback_id,omitempty"`
}