qujp( `dZddlZddlZddlmZddlmZddlmZddl m Z m Z m Z m Z mZddlmZmZmZddlmZdd lmZej.eZd Zd Zd ed efdZ dd edededed e f dZ!d eded e fdZ"dd eded efdZ#d ededed efdZ$d eded e fdZ%d ede d efdZ&d eded e fdZ'y)aThe tenant's uploaded documents, as the settings page and the chat see them. A document is an ingested file: its text lives in the knowledge base so the bot can answer from it, and the original file sits in blob storage so the bot can hand the actual PDF to a visitor who asks for it. Both facts come from the same place -- the metadata on the file's knowledge base rows -- so there is no separate documents table to keep in sync. The category is one more key in that metadata. Links given to visitors point at this service, not at blob storage, and never expire -- a link left in a chat transcript still works months later. The real storage link is a SAS token minted when the visitor clicks, valid for LINK_HOURS, which keeps the storage account private and out of the chat payload. N)quote)sql)settings)delete_vector_data_by_sourceget_db_connectionget_source_ids_by_nameinsert_summaryinsert_vector_data) chunk_textgenerate_summaryget_embeddings)extract_text_from_bytes)storage_service Uncategorised tenant_idreturncN|jd|f|jdS)zSome tenants have a schema but were never ingested into. Checking the schema alone is not enough: querying a missing table raises and would log a stack trace on every call for those tenants. zzSELECT EXISTS (SELECT FROM information_schema.tables WHERE table_schema = %s AND table_name = 'strategist_knowledge_base')r)executefetchone)currs 5/var/www/html/strategist-ai/app/services/documents.py _table_existsr(s. KK P   <<>! uploadscategoryuser_idc :|xsdjxst}gd}}|D]$\}}t||dD]H}tj d|d|d|t || t j||d|Jt||} || d zz }ttj} || d|d } |r|| d <t| } tj d t| d|d| d| D]}t!|||}t#||||  t j$|| d|||j'| ||d't)|||}t+|d|||dS#t$r&} tjd|d| Yd } ~ ^d } ~ wwxYw)aIndex the uploaded files and store the originals. `uploads` is a list of (file_name, content_bytes). Re-uploading a name replaces the previous version, chunks and blob alike, so the tenant does not end up with two copies answering the same question. Returns the generated summary and a row per stored document, so a caller can show what was added without a second listing request. filezReplacing existing file  (z) in _z"Could not delete the old blob for : N )source source_idingestion_typerrzCreated z chunks for ))rr)r& file_namerfiles)summary documents)stripDEFAULT_CATEGORYrloggerinforr delete_file Exceptionwarningrstruuiduuid4r lenr r upload_fileappendr r )rrrrstoredall_textr)contentold_idextextr&metadatachunkschunk embeddingr+s ringest_documentsrD6sB%%';+;H2HF%. 7,Y 6J WF KK29+RxuYKX Y (F ; W++I&9+7NO  W'w :D4K % 'i&,(D ")HY D! hs6{m< {"YKqQR FE&u 7SI y%H E F ##I)Ai[/I7S II#+- .3.8x9gNG9gw/V 441 W!CI;bQSPTUVV Ws%E++ F4FFr&cZtfdt|Dd}|siS tj|d|dt|}t jd|dd|d ||d|d S#t$r(}t j d|dd|Yd}~bd}~wwxYw) aRemove a document's knowledge base rows and its stored file. The file name is looked up here rather than asked of the caller: the generic delete leaves the blob behind when the caller omits it, which quietly accumulates orphaned files nobody can reach. c34K|]}|dk(s |ywr&N.0dr&s r z"delete_document..lsV!K.I:U!Vs Nr"r)zCould not delete the blob for r#zDeleted document r!z chunks) from )r&r)deleted_chunks) nextlist_documentsrr1r2r/r3rr0)rr&matchr>deleteds ` rdelete_documentrRes V^I6VX\ ]E  T##I)AeK>P=Q/RS +9i@G KK#E+$6#7r'.QZP[\]"{1C% '' T7k8J7K2bTRSSTsA99 B*B%%B*c .t} |j5}t||sgcddd|jSt j dj t j|}|j|g}|jD]b\}}}} |r|s |xst}|r"|j|jk7r;|j|||| r| jnddd|cddd|jS#1swYnxYwnE#t$r9} tj!d|d| dgcYd} ~ |jSd} ~ wwxYw |jy#|jwxYw)z1Every uploaded file for the tenant, newest first.Nak SELECT metadata->>'source_id', metadata->>'source', metadata->>'category', MIN(created_at) FROM {}.strategist_knowledge_base WHERE metadata->>'ingestion_type' = 'file' GROUP BY 1, 2, 3 ORDER BY 4 DESC )r&r)r uploaded_atzCould not list documents for r#Texc_info)rcursorrcloserSQLformat Identifierrfetchallr.lowerr9 isoformatr2r/error) rrconnrqueryr,r&namecatcreatedr>s rrOrO|s  D! [[] c i0  @ ;GG  y12  KK I14 - 4g -- x~~/? ?  !*!% #:A7#4#4#6t " 7  @ A   8  4YKr"FQU V  9 @  sXD+D D+CD D+D'#D+*F+ E-4E(E-F(E--FFc |xsdjxst}t} |j5}t ||s ddd|j yt jdjt j|}|j|||f|j}ddd|jtjd|dd|d|d |d kD|j S#1swYPxYw#t$rG}|j!tj#d |d |d Yd}~|j yd}~wwxYw#|j wxYw)zGRe-categorise a document. Every chunk of the file carries the metadata.rNFz UPDATE {}.strategist_knowledge_base SET metadata = jsonb_set(metadata::jsonb, '{{category}}', to_jsonb(%s::text)) WHERE metadata->>'source_id' = %s zSet category 'z' on z chunks of r!r(rzCould not set category for r#TrU)r-r.rrWrrXrrYrZr[rrowcountcommitr/r0r2rollbackr_)rr&rr`rrachangedr>s r set_categoryrjsGB%%';+;H  D [[] #c i0 #$ GGy12  KK)4 5llG #  nXJeG9K {RTU^T__`ab{ % # #   29+RtDt T  sMDDDAD6>DD D E!-EE$E!!E$$E6cHsiStfdt|DiS)zBOne document by its id, or {} if this tenant has no such document.c32K|]}|dk(r|ywrGrHrIs rrLz find_document..s$1qkNi/1s)rNrO)rr&s `r find_documentrms,  1N95124 66rdocumentcV|sytj||dd|dtS)zA short-lived direct link to the stored file. Minted at the moment someone asks for it, never handed out in advance, so the window in which it is useful to anyone who intercepts it is small. rr&r"r))hours)r download_url LINK_HOURS)rrns rblob_urlrss=   ' 'h{+,Ah{.C-DEZ YYrr)c |xsdjjsiSt|}tfd|Dd}|sn|Dcgc]}|djvs|}}t |dk(r|d}n5t |dkDr't j |dt |d|d |siS|d|d |d tjjd d t|dt|d dScc}w)aResolve a file name to a permanent link the visitor can be given. The chat model knows file names because search results carry them as `source`, so that is what it passes here. The returned URL points at this service, not at blob storage. It does not expire, so a link sitting in a chat transcript still works next month; the real storage link is generated when the visitor actually clicks. rc3PK|]}|djk(s|yw)r)N)r])rJrKwanteds rrLz download_link..s%K!K.*>*>*@F*J!Ks&&Nr)rz matches z documents for z; refusing to guess.rr&/z/documents/download?tenant_id=z &source_id=)r)rr&url) r-r]rOrNr7r/r0rPUBLIC_BASE_URLrstripr)rr)r,rPrK candidatesrvs @r download_linkr}s/o2 $ $ & , , .F  y)I KYKT RE  "+OAf+8L8L8N.NaO O z?a qME _q KK9-yZ0A$+%9; <  {+j){+//66s;<=""' "2!3;uU;EW?X>Y[ ]]Ps D)D)NN)N)(__doc__loggingr5 urllib.parserpsycopg2rapp.core.configrapp.services.databaserrrr r app.services.embeddingr r r app.services.extractionrapp.services.storager getLogger__name__r/rrr.r4boolrlistdictrDrRrOrjrmrsr}rHrrrsG $PO;0   8 $  # # $ EI$(,5,5d,5c,5!,5-1,5^'s's't'.$c$S$D$NCC3446S6S6T6 Y Yt Y Y']S']S']T']r