mysql database, with a column to store the embeddings vector (yeah i'm not so fancy to be using pinecone :P)

To keep the "memory", do you pass the embeddings along with the new text prompt in an API call? How do you combine embeddings and text prompts? I don't know much about this, sorry if the question sounds silly.

use llama index: https://gpt-index.readthedocs.io/en/latest/index.html

The below code takes a list of questions from an excel, and answers each one based on the directory I passed in. I use this for answering Statement of Works for proposals i write as a first path. Usually, I will have a number of different directorys that i pass in to 'Talk' to different intellegences and get a couple different answers for each prompt. One trains on the entire corpus of my past performance. One has a simple document discussing tone and other information, and one in training on only the SOW itself.

   def excelGPT(dir, excel_file, sheet):
    #my GPT Key
    os.environ['OPENAI_API_KEY'] = 'sk-~Your open AI Key Here'
    #Working Directory for training
    root_folder = ''
    documents = SimpleDirectoryReader(root_folder).load_data()
    index = GPTSimpleVectorIndex(documents)
    file_name = dir + excel_file
    

    df = pd.read_excel(file_name, sheet_name=sheet)

    answer_array = []
    df_series = df.iloc[:,0]
    
    for i,x in enumerate(df_series):
        print("This is the index ", i)
        print(x)
        response = index.query(x)
        answer_array.append(str(response))

    zip_to_doc(df_series, answer_array, dir)

Hey, is it alright if you explain this in a bit more detail. I've playing around with llama-index myself. Do you have multiple indices? Or do you run each question through and get multiple responses. Isn't that quite expensive?

How do you also deal with the formatting of the various excel files. Would love to see the source code for this if you are willing to share?

I explain it here: https://www.youtube.com/watch?v=JHkdK_9ZnFw&list=PLhZ4HC__ic... (its a bit old, I've made quite a few changes, but the essence is the same)

which column type do you use?