beansack

package
v0.0.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 6, 2024 License: MIT Imports: 6 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	CHANNEL = "social media channel"
	POST    = "social media post"
	ARTICLE = "news article"
	COMMENT = "social media comment"
	INVALID = "__INVALID__"
)
View Source 
const (
	BEANSACK    = "beansack"
	BEANS       = "beans"
	NOISES      = "noises"
	KEYWORDS    = "keywords"
	NEWSNUGGETS = "concepts"
)

Variables

This section is empty.

Functions

func AddBeans 

func AddBeans(beans []Bean)

Adding feeds from news sources and social media Steps:

  1. Filter out the tiny ones for now
  2. Truncate the contents to keep below the limit
  3. Add the beans to the database
  4. Add media noise to database
  5. Create news nuggets and add to db
  6. Create embeddings for news nuggets and add to db
  7. Create generated fields for the beans and add them to database
  8. Map the news nuggets to the beans

func Cleanup 

func Cleanup(delete_window int)

func Equals 

func Equals(a, b *Bean) bool

func InitializeBeanSack 

func InitializeBeanSack(db_conn_str, emb_url string, emb_ctx int, pb_auth_token string) error

func Rectify 

func Rectify()

this is for any recurring service this is currently not being run as a recurring service

Types

type Bean 

type Bean struct {
	Url         string               `json:"url,omitempty" bson:"url,omitempty"`         // this is unique across each item regardless of the source and will be used as ID
	Updated     int64                `json:"updated,omitempty" bson:"updated,omitempty"` // date of update of the post or comment. Empty for subreddits
	Source      string               `json:"source,omitempty" bson:"source,omitempty"`   // which social media source is this coming from
	Title       string               `json:"title,omitempty" bson:"title,omitempty"`     // represents text title of the item. Applies to subreddits and posts but not comments
	Kind        string               `json:"kind,omitempty" bson:"kind,omitempty"`
	Text        string               `json:"text,omitempty" bson:"text,omitempty"`
	Author      string               `json:"author,omitempty" bson:"author,omitempty"`   // author of posts or comments. Empty for subreddits
	Created     int64                `json:"created,omitempty" bson:"created,omitempty"` // date of creation of the post or comment. Empty for subreddits
	*MediaNoise `bson:"-,omitempty"` // don't serialize this for BSON

	Keywords           []string  `json:"keywords,omitempty" bson:"keywords,omitempty"`                       // This can come from input and/or computed from a small language model
	Summary            string    `json:"summary,omitempty" bson:"summary,omitempty"`                         // generated from a large language model
	Topic              string    `json:"topic,omitempty" bson:"topic,omitempty"`                             // generated from a large language model
	SearchEmbeddings   []float32 `json:"search_embeddings,omitempty" bson:"search_embeddings,omitempty"`     // generated from a large language model
	CategoryEmbeddings []float32 `json:"category_embeddings,omitempty" bson:"category_embeddings,omitempty"` // generated from a large language model
	SearchScore        float64   `json:"search_score,omitempty" bson:"search_score,omitempty"`               // generated from DB search algorithm
}

func FuzzySearch 

func FuzzySearch(options *SearchOptions) []Bean

Searches beans based on search options Algorithm:

  1. Look for category embeddings in input. If so, category search with that
  2. If NO category embeddings are found then create embeddings from category texts and search with those
  3. If NO category texts are found then create embeddings from the conversational context and search with that 3.ALT. If context search does not return a value to a TextSearch
  4. If NO vector input is available just do a regular search

func NuggetSearch 

func NuggetSearch(nuggets []string, settings *SearchOptions) []Bean

func Retrieve 

func Retrieve(options *SearchOptions) []Bean

This retrieves beans using scalar filter instead of fuzzy searching

func TextSearch 

func TextSearch(keywords []string, settings *SearchOptions) []Bean

func TrendingBeans 

func TrendingBeans(options *SearchOptions) []Bean

Returns the trending news/posts defined by the search parameter such as: by the day/week, by category match Algorithm:

  1. Find all the news/posts for that day that matches the categories (match everything if there is no category)
  2. Find the nuggets that are mapped to these articles
  3. Take the highest nugget trend score and assign to the respective article
  4. Stack rank the news/posts by that trend score

type BeanNugget 

type BeanNugget struct {
	ID          any       `json:"_id,omitempty" bson:"_id,omitempty"`
	KeyPhrase   string    `` /* 294-byte string literal not displayed */
	Event       string    `` /* 249-byte string literal not displayed */
	Description string    `` /* 135-byte string literal not displayed */
	Embeddings  []float32 `json:"-,omitempty" bson:"embeddings,omitempty"`
	Updated     int64     `json:"updated,omitempty" bson:"updated,omitempty"`
	TrendScore  int       `json:"match_count,omitempty" bson:"match_count,omitempty"`
	BeanUrls    []string  `json:"mapped_urls,omitempty" bson:"mapped_urls,omitempty"`
}

func TrendingNuggets 

func TrendingNuggets(options *SearchOptions) []BeanNugget

Finds the trending news nuggets defined by the search parameter such as: by the day/week, by category match Algorithm:

  1. (Optional) Find all the nuggets in that day/week and get their urls
  2. Match all the beans irrespective of updated: 0/1 within the category match threshold
  3. Find the nuggets that has those URLs as mapped urls for that day
  4. Stack rank them by trend score

type BeanSackError 

type BeanSackError string

func (BeanSackError) Error 

func (err BeanSackError) Error() string

type MediaNoise 

type MediaNoise struct {
	BeanUrl       string  `json:"mapped_url,omitempty" bson:"mapped_url,omitempty"` // the id is 1:1 mapping with Bean.Id
	Updated       int64   `json:"updated,omitempty" bson:"updated,omitempty"`
	Source        string  `json:"source,omitempty" bson:"source,omitempty"` // which social media source is this coming from
	ContentId     string  `json:"cid,omitempty" bson:"cid,omitempty"`       // unique id across Source
	Name          string  `json:"name,omitempty" bson:"name,omitempty"`
	Channel       string  `json:"channel,omitempty" bson:"channel,omitempty"` // fancy name of the channel represented by the channel itself or the channel where the post/comment is
	ContainerUrl  string  `json:"container_url,omitempty" bson:"container_url,omitempty"`
	Comments      int     `json:"comments,omitempty" bson:"comments,omitempty"`       // Number of comments to a post or a comment. Doesn't apply to subreddit
	Subscribers   int     `json:"subscribers,omitempty" bson:"subscribers,omitempty"` // Number of subscribers to a channel (subreddit). Doesn't apply to posts or comments
	ThumbsupCount int     `json:"likes,omitempty" bson:"likes,omitempty"`             // number of likes, claps, thumbs-up
	ThumbsupRatio float64 `json:"likes_ratio,omitempty" bson:"likes_ratio,omitempty"` // Applies to subreddit posts and comments. Doesn't apply to subreddits
	Score         int     `json:"score,omitempty" bson:"score,omitempty"`
	Digest        string  `json:"digest,omitempty" bson:"digest,omitempty"`
}

func (*MediaNoise) PointsTo 

func (n *MediaNoise) PointsTo(a *Bean) bool

type SearchOptions 

type SearchOptions struct {
	ScalarFilter     store.JSON
	TopN             int
	SearchTexts      []string
	SearchEmbeddings [][]float32
	Context          string
}

func NewSearchOptions 

func NewSearchOptions() *SearchOptions

func (*SearchOptions) WithKind 

func (settings *SearchOptions) WithKind(kinds []string) *SearchOptions

func (*SearchOptions) WithTimeWindow 

func (settings *SearchOptions) WithTimeWindow(time_window int) *SearchOptions

func (*SearchOptions) WithTopN 

func (settings *SearchOptions) WithTopN(topn int) *SearchOptions

func (*SearchOptions) WithURLs 

func (settings *SearchOptions) WithURLs(urls []string) *SearchOptions

type Sip 

type Sip struct {
	Username string `json:"username,omitempty" bson:"username,omitempty"`
	Source   string `json:"source,omitempty" bson:"source,omitempty"`
	BeanUrl  string `json:"url,omitempty" bson:"url,omitempty"` // the id is 1:1 mapping with Bean.Id
	Action   string `json:"action,omitempty" bson:"action,omitempty"`
}

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.
⌂ 换网址