j"dZddlZddlZddlmZmZmZmZddlm Z ddl m Z m Z ddl mZddlmZddlmZmZmZmZmZmZdd lmZdd lmZmZej:eZed d g Z de!de"de#fdZ$e jKdedfde"fdZ&e jKdedfde"fdZ'e jQdedfde!fdZ)e jUdedfde!de!de!fdZ+y)z-Connect and manage product ingestion sources.N) APIRouterBody HTTPExceptionQuery) start_job)get_integrationset_integration_status)settings build_catalog) KIND_CRAWL KIND_HTTP_API KIND_SHOPIFY get_sourcesmark_source_status upsert_source)jobs)UnsafeUrlErrorassert_safe_urlz/sourcessources)prefixtagskindconfigreturnc`g}|tk(r"|jds|jd|S)aOWhat must still be supplied before this source's products price correctly. A source with no currency still ingests -- its products are simply flagged incomplete and excluded from serving, which from the API looks identical to an empty catalogue. This is the surfaced prompt that replaces guessing: the merchant can see exactly what's missing instead of silently getting nothing recommended. KIND_CRAWL is excluded here even though its config can also lack a currency: fetch_products() resolves it at crawl time via an LLM inference over page context (see crawl.py's _infer_currency) when no page's own markup states one, so there is nothing for the merchant to supply. KIND_HTTP_API has no such fallback -- an API's JSON has no equivalent of schema.org markup to infer from, so it still needs it. currency)rgetappend)rrneedss ./var/www/html/strategist-ai/app/api/sources.py_needsr"s- E }VZZ %; Z Lz/http.payloadc K|jd}|jd}|r|s tdd t|t j |j}||jdxsd |jd xsd d d ddd|jdxsi|jd|jd|jdxsddid}|jdrd|dini}t|t|||t|ddtt|dS#t$r}tdt |d}~wwxYww)aConnect a custom product API. Only a base URL and credentials are required. The field map and url_template are inferred by the pipeline on first sync -- a merchant should not have to describe their own API's shape. Currency is not inferred or assumed from anything else the tenant has connected (a wrong guess there silently mis-prices a whole catalog); it is either supplied here or reported back via "needs" so the merchant knows to set it. This endpoint never refuses a connection for any of their absence. tenant_idbase_urlz#tenant_id and base_url are required status_codedetailN records_pathz $.products paginationoffsetlimitskipz$.total)style limit_param offset_param page_size total_pathfields url_templaterauthr3none)r'r-r.r8r9rr:api_keyactivepending_first_sync)r external_refstatusmappingr ) rrrrstrhttpxURLhostrrr")r$r&r'exr?r credentialss r!create_http_sourcerH-sP K(I{{:&H H#HJ J=!99X&++L N3C|kk,/Ng"94N ++h'-2 N3KK + F#8'8 F6=[[5K9gi01QSK)]L&+N!<+?M62 44) =CG<<=s)4E  D$C"E $ E-EEE z/websitecnK|jd}|jdxs|jd}|r|s tdd t|t j |j}||jd |jd xstj|jd d }t|t||it|d dtt|d}|jddr$dd} t|d|d{|d<d|d<|S|S#t$r}tdt |d}~wwxYw77#tj$r}|j |d<d|d<Yd}~|Sd}~wt$r1}t"j%d||j&d|d<Yd}~|Sd}~wwxYww)a[Connect the merchant's own website, read by crawling it. The third way in, for a merchant with neither a spreadsheet nor an API: they give us a page on their site and we read the products off it. Prices come from the page's schema.org markup where it has any -- data the merchant already publishes for search engines, so it is exact rather than inferred. A page without it still yields a product, flagged as having no price. currency does not need to be supplied here: if no page states one in its own markup, fetch_products() infers it from page context (domain, locale, shipping/tax text) with a dedicated LLM call at crawl time, rather than asking the caller for it up front. Saving the connection also starts a build (sync -> enrich -> pair) for the whole tenant, not just this source -- pairing has to see every product to find cross-product relationships, so there is no such thing as pairing scoped to one newly added source. Every OTHER already-connected source still gets its normal 24h re-crawl throttle in that build; this one is always crawled fresh since it has never been read before. Set "auto_build": false to skip this and connect without triggering anything (e.g. a caller adding several sources in one batch that wants a single build at the end instead of one per source). r&urlr'r(ztenant_id and url are requiredr)r,Nr max_pagesmin_interval_hours)rJrrKrLr=pending_first_crawl)rr?r@productsr auto_buildTct||S)N)progressr )tenantrQs r!runz"create_website_source..runs (; ;r#buildjob_idqueued job_status#already_running_without_this_sourcez2Could not auto-start build after connecting %s: %s not_started)N)rrrrrBrCrDrEr CRAWL_MAX_PAGESrr r"rrJobAlreadyRunningrUloggerwarningr+) r$r&rJrFr?rresponserSrunnings r!create_website_sourcer`Zs2 K(I ++e  7 J 7C C#CE E=99S>&&LKK +[[-I1I1I&kk*>? F)ZvrB"L"0E F35H{{<& < 3'0GS'I!IHX %-H\ " O8OQ =CG<<=4"J%% K ")HX %JH\ " O  3 NNO' 4%2H\ " O  3syAF5 DB(F5>EE EF5 E(D>>EF5EF2E5/F55 F2&F-'F5-F22F5r&cKt|Dcgc].}|d|d|d|d|d|dt|d|dd0}}t|t}|d|r|did Sd id Scc}ww) z6Every source for a tenant. Never includes credentials.rr?rr@ connected_atlast_synced_at)rr?rr@rcrdr shopifyN)rplatform_integration)rr"rr)r&sr integrations r! list_sourcesrisY'   6An,=X;!H+>*a@P>Q6AhK0 2G")\:KK;x0%K LLEI%K LLsA+3A&(A+z/{kind}/{external_ref}r?cKt|Dcgc]}|d|k(r |d|k(r|}}|s tddt||d|tk(rt |td||ddScc}ww) Nrr?izNo such connected sourcer)revokedINACTIVE)rr?r@)rrrrr )rr?r&rgmatchess r! delete_sourcerns%i0KQF)t#.(9\(IKGK 4NOOt\95 | y, C,) LLKsA,A'AA,),__doc__loggingrCfastapirrrrapp.api.catalogr"app.services.integrations.platformrr app.core.configr app.services.catalog.buildr app.services.catalog.sourcesr rrrrrapp.services.infrarapp.services.infra.ssrfrr getLogger__name__r\routerrBdictlistr"postrHr`rrideleternr#r!rs33 99%V$4$C   8 $ *I; 7dt,W-1#Y)4d)4)4XZ04S HHHVB(-c L# L L '(GLSz Mc M M M) Mr#