📄️ Text Encoding with Bi-Encoder
bi_enc is a arrangement of two encoder modules from BERT, it represents context and candidate separately using twin-structured encoders, it takes contexts and candidates, to predict the best suitable candidate for each context. You can train the module on custom data to behave accordingly. Let's take a deep dive into the training culture.
📄️ Text Summerization with Sumy Summarizer
Module cl_summer uses the sumy summarizer to extract summary from text.
📄️ **Dolly V2 (dolly)**
Databricks’ dolly-v2, an instruction-following large language model trained on the Databricks machine learning platform that is licensed for commercial use. Based on pythia-2.8b, Dolly is trained on ~15k instruction/response fine tuning records databricks-dolly-15k generated by Databricks employees in capability domains from the InstructGPT paper, including brainstorming, classification, closed QA, generation, information extraction, open QA and summarization. dolly-v2-3b is not a state-of-the-art model, but does exhibit surprisingly high quality instruction following behavior not characteristic of the foundation model on which it is based.
📄️ Text Generation with Chat GPT2
Module gp2 uses the OpenAI's GPT-2-medium to perform text generation on a given text.
📄️ **Large Language Models (llm)**
_NOTE:_ This module only works with CUDA enabled devices. If you run into any bitsandbytes error, try checking the CUDA version of your device and replace the libbitsandbytescpu.so with libbitsandbytescuda.so in /home//anaconda3/envs/llm/lib/python3.10/site-packages/bitsandbytes/ directory.
📄️ Text Paraphrasing
Module paraphraser implemented for paraphrasing the input sentence. Paraphrasing is the act of expressing the same meaning of a written or spoken text using different words, without changing the original message or altering the main ideas. Paraphrasing is often used to restate a passage in a simpler or more concise way, to avoid plagiarism, or to clarify complex ideas. It involves understanding the original text, analyzing its key points, and rewording it in your own words while maintaining the original meaning. Effective paraphrasing requires good language skills, attention to context, and a clear understanding of the purpose of the text.
📄️ Text Similarity with Sentence BERT Model
sbert_sim is a implementation of SentenceBert for scoring similarity among sentences, it uses bi-encoder in a saimese setup to encode the sentence followed by the cosine similarity to score the similarity.
📄️ Sentiment Analysis
Sentiment analysis is a technique used in natural language processing and machine learning to automatically identify and extract subjective information from text, such as opinions, emotions, and attitudes. It involves analyzing large volumes of text data to determine whether the overall sentiment expressed is positive, negative, or neutral.
📄️ Text Summarization
Module summarization uses the philschmid/bart-large-cnn-samsum to get the abstractive summary of a text.
📄️ Text Segmentation
Module text_seg implemented for the Topical Change Detection in Documents via Embeddings of Long Sequences.
📄️ Entity Extraction with Transformers
What is NER: named entity recognition (NER) — sometimes referred to as entity chunking, extraction, or identification — is the task of identifying and categorizing key information (entities) in text. An entity can be any word or series of words that consistently refers to the same thing. Every detected entity is classified into a predetermined category.
📄️ Topic Extraction
Module topicext implemented for extracting topics from set of documents. This module accepts as input a set of documents and their associated class labels. You may use Jaseci cluster module to cluster document list to similar groups. This is an example code to demonstrate topicext module.
📄️ Text Encoding with USE Encoder
Module use_enc uses the universal sentence encoder to generate sentence level embeddings. The sentence level embeddings can then be used to calculate the similarity between two given text via cosine similarity and/or dot product.
📄️ Text Encoding with USE QA Model
Module useqa uses the multilingual-qa to generate sentence level embeddings. The sentence level embeddings can then be used to calculate best match between question and available answers via cosine similarity and/or distscore.