j"1 dZddlZddlZddlZddlmcmZddlm Z ddl Z ddl m Z ddl mZddlmZddlmZmZddlmZmZdd lmZdd lmZej8eZd Zd Z d e!de"fdZ#dZ$Gdde%Z&de!fdZ'de(de(fdZ)de(fdZ*de(de+fdZ,de(de!fdZ-d#de(de!de!de(fdZ.de"de!de!fd Z/d#d!e(de!de!de"fd"Z0y)$aTurns a crawled storefront into source products. The third way into the catalogue, alongside a CSV upload and a product API: the merchant gives us a page on their own site and we read it. The pages themselves are fetched and parsed by the website crawler that already serves chat ingestion; this module's job is to hand what it finds to normalise() in exactly the shape the Shopify and HTTP connectors produce, so a crawled product is a first-class catalogue row rather than a name and a picture. Price, brand and stock come from the page's own schema.org markup where it has any. That is exact data the merchant already publishes for Google, so reading it costs nothing and cannot hallucinate. A page without it yields a product with no price, which normalise() flags -- the same treatment a CSV row with no currency gets, and for the same reason: a guessed price is worse than a missing one. N)urlparse)settings)client)CURRENCY_INFERENCE_PROMPT)backfill_categoriesextract_page_products)_looks_like_product_urlextract_text_from_url)insert_llm_usage)assert_safe_urliz-{http://www.sitemaps.org/schemas/sitemap/0.9}urlreturncK t|}tjd4d{}|j|jd|j dd{}|j dk7rgcdddd{Stj|j}|jtdk(r|jtdDcgc]}|js|j}}td |Dd}|sgcdddd{S|j|d{}|j dk7rgcdddd{Stj|j}|jtdDcgc]}|js|j}}dddd{t%} g} D]5} t'| s| | vs| j)| | j+| 7| S777cc}w777cc}w7f#1d{7swYvxYw#t$r#} t j#d || gcYd} ~ Sd} ~ wwxYww) zProduct URLs straight from /sitemap.xml, cheaper than discovering them by crawling. Never raises -- any failure just falls back to the normal link-following crawl.g$@)timeoutNz://z /sitemap.xml sitemapindexlocc3FK|]}d|jvs|yw)productN)lower).0cs 9/var/www/html/strategist-ai/app/services/catalog/crawl.py z(_discover_via_sitemap..4s&UQi1779>Tq&Us!!z$Sitemap discovery skipped for %s: %s)rhttpx AsyncClientgetschemenetloc status_codeET fromstringcontenttag _SITEMAP_NSitertextnext Exceptionloggerinfosetr addappend) r baserresprootelchildrenproducts_childurlsexseen product_urlsus r_discover_via_sitemapr:&s:}$$T2 Q Qft{{m3t{{m<$PQQD3& Q Q Q==.Dxxk],77.2ii;-s8K.LXPRPWPWBGGXX!%&U(&UW[!\% Q Q Q$ZZ77##s* Q Q Q}}T\\2&*ii;-s0C&DPBGGPDP! Q Q* 5DL # "1 %!4- HHQK    "# 7 QQ QY Q8 Q Q! Q Q Q Q"  :CD sFI*%H;H H;/H&H H&1 H;=H>H;I*AH&H$H2H& H;HH;I*H&0H1H& H;HH;I*9H&H"H0H&2 H;=H$>H;I*I*$&I* H; H&H;H&H;H&H;H&$H;&H8,H/ -H84H;; I'I"I'I*"I''I*) outofstocksoldout discontinuedceZdZdZy)CrawlSourceErrorzIThe crawl could not complete. Row-level problems are skipped, not raised.N)__name__ __module__ __qualname____doc__rr?r?PsSrEr?ct|tr |r|dnd}t|tr$|jdxs|jd}|dvrt |j SdS)zGschema.org lets almost any field be a string, a list, or a nested node.rNnamez@valueNrI) isinstancelistdictrstrstrip)values r_textrPTs^%!at% &!8UYYx%8!&j!83u:   @b@rEblockc|jd}|s7|jd}t|tr|r|djd}t|tr |r|dnd}t|tr|SiS)a;The offer to read price/availability from -- a plain Product's own offers, or (Shopify's ProductGroup schema) the first variant's offers when the group itself carries none. One variant's price stands in for the whole product here, matching to_source_product's single-variant model of a crawled page.offers hasVariantrN)rrJrKrL)rQrSvariantss r_offerrV]sl YYx F 99\* h %(a[__X.F&$$$-6525rEofferc|jdxs|jd}|dvry tt|jdd}|dk\r|SdS#tt f$rYywxYw)NpricelowPricerH,rIr)rfloatrMreplace TypeError ValueError)rWrawrYs r_priceramsu ))G  5 * 5C jc#h&&sB/0QJ5(D( z "s$AA*)A*ct|jdjtfdtD S)N availabilityc3&K|]}|v ywNrD)rtermrcs rrz_in_stock..zsBD4<'Bs)rPrrany _OUT_OF_STOCK)rWrcs @r _in_stockrixs3>2399;LBMBB BBrErc|jdxsi}|jd}|r t|S|jdxsdjdS)zA stable id for this product within its site. The page URL, not the name: a merchant renaming a product must update the row rather than orphan it and create a second one. r`sku product_urlrI/)rrMrstrip)rr`rks r _external_idro}sN ++e  "C ''%.C3s8N]!;!Ar I I# NNrE source_refcurrencyc >|jdxsijdxsi}t|}t|xs|jd}t|}|r|jdsyid|d|jdd|jdxsdd t |jd xs|jd xsdd dd |jd xsdd |jd xsdd dd|jdxsddgdgddddd|jdxsgd|td|jdit |idgdgdd|idddt |jdj xs|dS)ajOne crawled product in the shape normalise() consumes. price and brand prefer the page's own structured markup; product.get(...) is only ever consulted when that markup had nothing (extract_with_llm's price/brand are themselves None on any page that had JSON-LD or Open Graph data, since extract_page_products never reaches the model then). r`jsonldrYrGN external_idtitle descriptionbrandhandlerl image_url raw_category product_typecategorytags collectionsstatustracks_inventoryTctasrU highPrice)rYcompare_at_price availableoptionsattributes_raw source_kindcrawl priceCurrency)rptenant_relationsrating review_count featured_rankrq)rrVrarorPriupper)rrprqrQrWrYrts rto_source_productrs[[  %2 * *8 4 :E 5ME 5M 1W[[1Ew'K gkk&1' {' V$'  w{{=19T'  uyy)*Jgkk'.BJd '  $ '  w{{=19T '  W[[-5'  '   J/74'  '  r'  $'  D' "  F#)r#' *  &;1G'H I"5)   +' 6 "7' 8 w9' :! %))O45;;=IM' 'rEpages tenant_idcn KdjfdddDj}|sytjdj dxsd| fd } t j |d{}tj|jdjj} t|d d}|r.sC+f +-a.KsSTVYZ_V`Oa.ab+s@type json_object)modelmessagesmax_completion_tokensrresponse_format) openai_clientchat completionscreater LLM_MODEL)promptsr_callz_infer_currency.._callsI!!--44$$%&9:"%#]3 5  rEzCurrency inference failed: usagezCurrency Inferencez(Could not log currency inference usage: rq)joinrNrformatrasyncio to_threadjsonloadschoicesmessager#r)r*warninggetattrr rr prompt_tokenscompletion_tokens total_tokensrJrLrMrr) rrr'rresponsepayloadr6rrqrs ` @r_infer_currencyrs| ;;+ %bq + ++057  & - -%(,,u2E2KRV WF  **511**X--a088@@A H'40  Y(P>P"00%2I2I5K]K] _ +5Wd*Cw{{:&H h $H (: >> #2 4RD9: HA"FGGHshA!F5&E>E?:E:A FAF5E F"E?:F5?FF5 F2F-(F5-F22F5configc XK|jdxs|jd}|s tdt|t|jdxstj }t |d{}|r tjdt|| |rt|d|||d||d{}nt|d ||| d{}|jd xsg} | std |dg} | D]7} t| | jdd|D]} | j| | f9t!| || D cgc]\} } |  }} } |jds&t#d|Dst%| |d{fd|DDcgc]}||}}tjd|t| t||S7c7%7#t$r*}td |d t|j|d}~wwxYwcc} } w7cc}ww)aCrawl the merchant's site and return every product found, as source products. Raises rather than returning a short list when the crawl itself fails: the caller deletes products it did not see this run, so a half-finished crawl must never be mistaken for a shrunken catalogue. progress, if given, is called as progress(percent, step) as pages are visited -- best-effort, see extract_text_from_url. r base_urlz source config must declare a url max_pagesNz0Sitemap discovery found %d product url(s) for %sr) max_depthrr seed_urlsprogress)rrrrz crawl of z failed: rz returned no pageshtmlrIrqc3K|]D}t|jdxsijdxsijdFyw)r`rsrN)rVr)rrs rrz!fetch_products..#sF 0$%!'e (:'?'?'I'OR P T TUd e 0sA A c38K|]}t|ywre)r)rrrqrps rrz!fetch_products..(sRq)!ZBRsz6Crawl of %s visited %d page(s) and found %d product(s))rr?r intrCRAWL_MAX_PAGESr:r*r+rr r)rr@rr.rrgr)rrprrr rrresultr6rpairspager_foundspsource_productsrqs ` @rfetch_productsrsF **U  5vzz*5C ABB CFJJ{+Gx/G/GHI+C00I F NC ) V 0qI#JY/(DDF1rY)!##F JJw  %2E 3%/ABCC E*,T488FB3GS *G LL'4 ) **y)', -!W -E -zz*%H C 0). 00( ::SER*b.*O* KKHSZ_!57 W1 D# V3%yb9J9J8KLMSUUV" . ;*sA7H*;G<&H*#G'<G!=G'G$G'A5H* H7H*H#H*% H%0/H*!G'$G'' H0%HH H*%H*re)1rCrrloggingxml.etree.ElementTreeetree ElementTreer! urllib.parserrapp.core.configrapp.core.llm_clientrrapp.core.promptsr'app.services.content.product_extractionrrapp.services.content.websiter r app.services.infra.databaser app.services.infra.ssrfr getLoggerr@r*rr%rMrKr:rhr)r?rPrLrVraboolrirorrrrDrErrs4 ""! $76^W83   8 $ $= !S!T!N: TyTACA 6$ 64 6 )$)CTCdC O$O3O7t777t7t))#)#)X??3?3?Z^?rE