j dZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z ddl mZddlmZddlmZddlmZmZdd lmZej2eZhd Zd Zd Zd ZdZ gdZ!dZ"dZ#dddddddddZ$ejJdZ&dZ'dZ(ejJddjSde(DzdzejTZ+de,fdZ-dodZ.dZ/ejJddjSde/DzdzejTZ0d Z1de,fd!Z2d"Z3dpd#e4d$e4de4fd%Z5d&e4de4fd'Z6d&e4de4fd(Z7d)Z8de4fd*Z9d+e4d,e4de4fd-Z:d.Z;d/Zde,fd2Z?d3e4de4fd4Z@de4fd5ZAd,e4de4fd6ZBd7e4d3e4deCfd8ZDd9eCdeCfd:ZEd7e4d3e4deCfd;ZFd#e4de,fd<ZGdqd=eCd>e4d?e4deCfd@ZHdqd7e4d3e4d>e4d?e4deCf dAZIdBZJdCZKejJdDejTZLejJdEejTZMejJdFejTZNdGe4de4fdHZOdIe4dJe4de,fdKZPdeCfdLZQde4fdMZRdeCfdNZSdeCfdOZTdPZUde,fdQZVdRZWdeCfdSZXd7e4deCfdTZYhdUZZdGe4de4fdVZ[d=eCd>e4de\fdWZ] dqd7e4d3e4d>e4d?e4de\f dXZ^deCfdYZ_dZZ`dpd3e4d[eade4fd\Zbdqd[ead7e4deCfd]Zcd^e4de4fd_Zdd`ZedeafdaZfd[ead7e4deCfdbZgdcZhdeifddZjdrd[eadee4dfe4d7e4deCf dgZk dpd[ead7e4dee4dfe4deCf dhZldiZmdjZndpdkeCdee4dfe4ddfdlZodpdmeCdee4dfe4ddfdnZpy)saTurn a crawled page into product records. Pure parsing: no database, no network. Two sources, in order of trust -- the page's own JSON-LD, then its Open Graph meta tags. A third source, a language model reading the page text, lives in Task 4 and is only reached when neither of these produces anything. CTAs are never guessed. They are read straight off the page's own markup via extract_ctas() -- exact, since a real button's URL is right there in the HTML, rather than a language model's approximation of it. N)partial) parse_qsl urlencodeurljoinurlparse urlunparse) BeautifulSoup)settings)client)CATEGORY_BACKFILL_PROMPTPRODUCT_EXTRACTION_PROMPT)insert_llm_usage>viewvisitdonatebuy_nowenquire book_demo subscribe add_to_cart)z javascript:zmailto:ztel:<)h1h2h3h4h5))r) add-to-cartz /cart/addz ?add_to_cartz add to cartz add to bagz add to basket)r)zbuy nowzbuy-nowz purchase now)r)z book a demoz book demozschedule a demozrequest a demo)r)r)r)r)r) z contact usz get in touchrenquiryzrequest a quotez get startedz start freez talk to usz talk to sales)r)z learn morez view detailszsee morez read moreexplore)navheaderfooteraside)rrrrrrrrz \s*\|+\s*) contactcartcheckoutrrbookdemorr)relatedupsellz cross-sell crosssellz also-bought recommendz you-may-alsoz\b(|c#FK|]}tj|ywNreescape.0ms F/var/www/html/strategist-ai/app/services/content/product_extraction.py r:xGqbiilG!z)\breturncdj|jdxsg}|d|jdxsdzz }|d|jdxsdzz }ttj |S)N classidzdata-section-type)joingetbool_RELATED_CONTAINER_REsearchtaghaystacks r9_is_related_containerrK|soxx(.B/H swwt}*++H sww239r::H %,,X6 77c|jtD]}|j|jddiD]}|j|jtD]}|jy)zRemoves page chrome (nav/header/footer/aside, role=navigation) and cross-sell/related-products rails from soup, in place. Shared by CTA and option extraction so both see the same notion of "the page's own content".role navigationattrsN)find_all _CHROME_TAGS decomposerK)soupchromer,s r9_decompose_chromerWsx -- ---v|&<-===!67rL)z product-mainzproduct-detailz product-infozproduct-singlesummaryz entry-summaryc#FK|]}tj|ywr2r3r6s r9r:r:r;r<)mainarticlecd|jdxsdvrydj|jdxsg}|d|jdxsdzz }|jdd }ttj |S) NProductitemtyperBTr?r@rA_-)rDrCreplacerE_PRODUCT_CONTAINER_RErGrHs r9_is_product_containerrcs~SWWZ(.B/xx(.B/H swwt}*++HS)H %,,X6 77rLc^|jt}||S|jtS)a2The product's own detail area, if the page marks one -- CTAs and options are read from inside it when found, which is what keeps category/navigation links (books.toscrape.com's "Books", "Poetry" ...) out without needing to know what a category link looks like. Tried in order: an element matching _is_product_container, then a generic
/
landmark, then None (the whole page). find() returns the first match in document order, which for nested markup is the outermost (BeautifulSoup visits parents before children). )findrc _PRODUCT_CONTAINER_FALLBACK_TAGS)rU containers r9_find_product_containerrhs/ /0I 995 66rLurlskuc|rdt|jStj|xsdjj j j dd}d|S)zA stable id so re-ingestion updates a product instead of duplicating it. Prefers the SKU, which survives a site restructure. Falls back to a hash of the URL, which does not, but is better than nothing. zsku:rBNzurl:)strstriphashlibsha1lowerencode hexdigest)rirjdigests r9product_key_forrusi  c#hnn&'(( \\39"++-335<<> ? I I KCR PF &?rLnamectjdd|xsdjjjd}|xsdS)z?A stable, readable fragment of a name for use as part of a key.z [^a-z0-9]+r`rBunnamed)r4subrnrq)rvslugs r9_slugifyr{s@ 66-tzr&8&8&:&@&@&B C I I# ND  9rLcdt|S)aA stable id for a product identified only by its name. The model reads prose, not a per-product page or a SKU, so the same product mentioned on several pages (a homepage teaser and its own page, say) must collapse onto one key rather than minting a new one per page it happens to be mentioned on. The products table is per-tenant (its own schema), so a name collision across tenants is not a concern here. zname:)r{)rvs r9product_key_for_namer}s8D>" ##rLc8t|tr |r|dSdS|S)z;JSON-LD lets almost any field be a value or a list of them.rN) isinstancelistvalues r9_firstrs"% uQx*d* LrLct|}t|tr7|jdxs$|jdxs|jd}|rt |j SdS)zBA field that may be a string, a list, or a nested {'@value': ...}.z@valuervrirB)rrdictrDrmrnrs r9_text_ofrsY 5ME% (#Luyy'8LEIIe.s8fv!8sr)rq _CTA_RULESany)rrcta_typeneedlesrJs @r9 _classify_ctarsHq &&(H'' 88 8O rLcp|jdxs$|jdxs|jdS)zBThe URL an element points at, from whichever attribute carries it.r formactionz data-href)rD)rIs r9_href_ofrs, 776? Kcggl3 Ksww{7KKrL)r product_idsingle_add_to_cart_button>sign up for updatessubscribe to newslettergosendrGsubmitclt|jdxsgvrytfdtDS)zWhether this button or its enclosing form carries evidence it is a real add-to-cart action: a hidden add-to-cart/product_id input, or WooCommerce's own single_add_to_cart_button class on the button.r@Tc3HK|]}jdd|iywinputrvrPNrer7 field_nameforms r9r:z-_form_has_product_identity..s+6yy(` resolves to when it has no href/formaction/data-href of its own -- its enclosing `
`'s action, falling back to the page URL when the form has none, which is what a browser does. The product-identifying hidden inputs the form carries (add-to-cart, product_id) are folded into the query string, so the result is an actionable URL and not just a link back to the page. This is the standard e-commerce add-to-cart pattern: a variable product's "Add to cart" button is a form submit, not a link, so without this it is invisible to CTA extraction entirely. But the same pattern also covers a page's search box or newsletter signup, which are not product actions -- so a generic verb with no product identity (see _form_has_product_identity) returns None here rather than a URL, keeping it out of CTA extraction altogether rather than relying on filter_ctas_for_product to catch it later. buttonNtyperrTrnactionrBrrvrPrkeep_blank_values)query)rvrDrnrq find_parentrget_text_GENERIC_FORM_VERBSrrreappendrrrextendr_replacer) rIr button_typer label_textrtarget extra_paramsrfieldrpartsrs r9_form_action_hrefrsv" xx8776?.h557==?Kh ??6 "D | %c4 0\\\-335;;= , ,hhx &B - - /F06WX^V ,X^F L*5  '&*)= >6;7#)r002    U 3 4 5  V E ekkT :E LL enn9U+;n< ==rLcz|jdxsdj}|r|}nl|jdxsdj}|r|}nD|jt}|r|j ddnd}|r|}n|j dd}dj |j }|dtS)aThe element's own words for a button, in order of how deliberately the author chose them for exactly this purpose. A plain "Add to cart" button's flattened text is fine. But a card-shaped anchor -- a whole product card wrapped in one , heading and blurb included -- flattens into a run-on sentence that reads as a bug when rendered on a button. aria-label and title exist precisely to give an element accessible/tooltip text distinct from its visual content, so they win when present. A nested heading is the next best proxy for "the name of the thing this points to." Only after all of those are absent do we fall back to the raw flattened text. Every candidate is still the site's own words -- this only chooses which of them to surface. aria-labelrBtitler?T) separatorrnN)rDrnre _HEADING_TAGSrrCsplit_MAX_LABEL_CHARS)rI aria_labelrrheading heading_texts r9 _label_ofrQs '','-2446J!'R..0 Ehh}-GJQ7++c+FWYL$ s$ ? HHU[[] #E "" ##rLcZ|xsd}tjd|}|jS)anStrip structural noise a raw label can carry that a visitor should never see: pipe-separated taxonomy terms mashed together by markup ("Erin Recommends|Clothing") collapse to a single space so they read as words rather than a raw delimiter. Cross-product carousel buttons are no longer handled by inspecting the label at all -- see _decompose_chrome / _RELATED_CONTAINER_MARKERS, which removes the whole related-products rail structurally before a candidate is even collected, regardless of what a given site's button happens to say ("Add to cart", "Add to basket", or anything else). rBr?)_SEPARATOR_RUN_REryrn)rs r9 _clean_labelrts+ KRE  ! !#u -E ;;=rLhtmlc t|xsdd}t|t|xs|}|jddg}g}t}|D]}t|xs t||} | s| j} | r| dk(s| jdrH| jjtrlt|} | sz t|xsd| } | r| |vrt!| | } |j#| | | d |j%| |S#t$r'}t j d|d|gcYd}~Sd}~wwxYw#t$rYwxYw) aI/