Sunday, June 8, 2025
News PouroverAI
Visit PourOver.AI
No Result
View All Result
  • Home
  • AI Tech
  • Business
  • Blockchain
  • Data Science & ML
  • Cloud & Programming
  • Automation
  • Front-Tech
  • Marketing
  • Home
  • AI Tech
  • Business
  • Blockchain
  • Data Science & ML
  • Cloud & Programming
  • Automation
  • Front-Tech
  • Marketing
News PouroverAI
No Result
View All Result

Build a Language Model on Your WhatsApp Chats | by Bernhard Pfann, CFA | Nov, 2023

November 22, 2023
in AI Technology
Reading Time: 3 mins read
0 0
A A
0
Share on FacebookShare on Twitter


To train a language model, we need to break language into pieces (so-called tokens) and feed them to the model incrementally. Tokenization can be performed on multiple levels.

Character-level: Text is perceived as a sequence of individual characters (including white spaces). This granular approach allows every possible word to be formed from a sequence of characters. However, it is more difficult to capture semantic relationships between words.

Word-level: Text is represented as a sequence of words. However, the model’s vocabulary is limited by the existing words in the training data.

It turned out that my training data has a vocabulary of ~70,000 unique words. However, as many words appear only once or twice, I decided to replace such rare words by a “<UNK>” special token. This had the effect of reducing vocabulary to ~25,000 words, which leads to a smaller model that needs to be trained later.

from nltk.tokenize import RegexpTokenizer

def custom_tokenizer(txt: str, spec_tokens: List[str], pattern: str=”|\\d|\\\\w+|[^\\\\s]\”) -> List[str]:\”\”\”Tokenize text into words or characters using NLTK\’s RegexpTokenizer, considerung given special combinations as single tokens.

:param txt: The corpus as a single string element.:param spec_tokens: A list of special tokens (e.g. ending, out-of-vocab).:param pattern: By default the corpus is tokenized on a word level (split by spaces).Numbers are considered single tokens.

>> note: The pattern for character level tokenization is \’|.\’\”\”\”pattern = \”|\”.join(spec_tokens) + patterntokenizer = RegexpTokenizer(pattern)tokens = tokenizer.tokenize(txt)return tokens


[“Alice:”, “Hi”, “how”, “are”, “you”, “guys”, “?”, “<END>”, “Tom:”, … ]
from collections import Counter

def get_infrequent_tokens(tokens: Union[List[str], str], min_count: int) -> List[str]:\”\”\”Identify tokens that appear less than a minimum count.

:param tokens: When it is the raw text in a string, frequencies are counted on character level.When it is the tokenized corpus as list, frequencies are counted on token level.:min_count: Threshold of occurence to flag a token.:return: List of tokens that appear infrequently. \”\”\”counts = Counter(tokens)infreq_tokens = set([k for k,v in counts.items() if v<=min_count])return infreq_tokens

def mask_tokens(tokens: List[str], mask: Set[str]) -> List[str]:\”\”\”Iterate through all tokens. Any token that is part of the set, is replaced by the unknown token.

:param tokens: The tokenized corpus.:param mask: Set of tokens that shall be masked in the corpus.:return: List of tokenized corpus after the masking operation.\”\”\”return [t.replace(t, unknown_token) if t in mask else t for t in tokens]

infreq_tokens = get_infrequent_tokens(tokens, min_count=2)tokens = mask_tokens(tokens, infreq_tokens)


[“Alice:”, “Hi”, “how”, “are”, “you”, “<UNK>”, “?”, “<END>”, “Tom:”, … ]



Source link

Tags: BernhardBuildCFAChatslanguagemodelNovPfannWhatsApp
Previous Post

Episode #509: Austin Root, Stansberry Asset Management – The Case For Productive Assets – Meb Faber Research

Next Post

Winning the cloud game: Phoning the right friend to answer the cloud optimization question

Related Posts

How insurance companies can use synthetic data to fight bias
AI Technology

How insurance companies can use synthetic data to fight bias

June 10, 2024
From Low-Level to High-Level Tasks: Scaling Fine-Tuning with the ANDROIDCONTROL Dataset
AI Technology

From Low-Level to High-Level Tasks: Scaling Fine-Tuning with the ANDROIDCONTROL Dataset

June 10, 2024
Decoding Decoder-Only Transformers: Insights from Google DeepMind’s Paper
AI Technology

Decoding Decoder-Only Transformers: Insights from Google DeepMind’s Paper

June 9, 2024
How Game Theory Can Make AI More Reliable
AI Technology

How Game Theory Can Make AI More Reliable

June 9, 2024
Buffer of Thoughts (BoT): A Novel Thought-Augmented Reasoning AI Approach for Enhancing Accuracy, Efficiency, and Robustness of LLMs
AI Technology

Buffer of Thoughts (BoT): A Novel Thought-Augmented Reasoning AI Approach for Enhancing Accuracy, Efficiency, and Robustness of LLMs

June 9, 2024
Deciphering Doubt: Navigating Uncertainty in LLM Responses
AI Technology

Deciphering Doubt: Navigating Uncertainty in LLM Responses

June 9, 2024
Next Post
Winning the cloud game: Phoning the right friend to answer the cloud optimization question

Winning the cloud game: Phoning the right friend to answer the cloud optimization question

ADA Website Compliance: Ensuring a Compliant Website

ADA Website Compliance: Ensuring a Compliant Website

Using Data Analysis to Improve and Verify the Customer Experience and Bad Reviews

Using Data Analysis to Improve and Verify the Customer Experience and Bad Reviews

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

  • Trending
  • Comments
  • Latest
23 Plagiarism Facts and Statistics to Analyze Latest Trends

23 Plagiarism Facts and Statistics to Analyze Latest Trends

June 4, 2024
Managing PDFs in Node.js with pdf-lib

Managing PDFs in Node.js with pdf-lib

November 16, 2023
Accenture creates a regulatory document authoring solution using AWS generative AI services

Accenture creates a regulatory document authoring solution using AWS generative AI services

February 6, 2024
Salesforce AI Introduces Moira: A Cutting-Edge Time Series Foundation Model Offering Universal Forecasting Capabilities

Salesforce AI Introduces Moira: A Cutting-Edge Time Series Foundation Model Offering Universal Forecasting Capabilities

April 3, 2024
The Importance of Choosing a Reliable Affiliate Network and Why Olavivo is Your Ideal Partner

The Importance of Choosing a Reliable Affiliate Network and Why Olavivo is Your Ideal Partner

October 30, 2023
Programming Language Tier List

Programming Language Tier List

November 9, 2023
Can You Guess What Percentage Of Their Wealth The Rich Keep In Cash?

Can You Guess What Percentage Of Their Wealth The Rich Keep In Cash?

June 10, 2024
AI Compared: Which Assistant Is the Best?

AI Compared: Which Assistant Is the Best?

June 10, 2024
How insurance companies can use synthetic data to fight bias

How insurance companies can use synthetic data to fight bias

June 10, 2024
5 SLA metrics you should be monitoring

5 SLA metrics you should be monitoring

June 10, 2024
From Low-Level to High-Level Tasks: Scaling Fine-Tuning with the ANDROIDCONTROL Dataset

From Low-Level to High-Level Tasks: Scaling Fine-Tuning with the ANDROIDCONTROL Dataset

June 10, 2024
UGRO Capital: Targeting to hit milestone of Rs 20,000 cr loan book in 8-10 quarters: Shachindra Nath

UGRO Capital: Targeting to hit milestone of Rs 20,000 cr loan book in 8-10 quarters: Shachindra Nath

June 10, 2024
Facebook Twitter LinkedIn Pinterest RSS
News PouroverAI

The latest news and updates about the AI Technology and Latest Tech Updates around the world... PouroverAI keeps you in the loop.

CATEGORIES

  • AI Technology
  • Automation
  • Blockchain
  • Business
  • Cloud & Programming
  • Data Science & ML
  • Digital Marketing
  • Front-Tech
  • Uncategorized

SITEMAP

  • Disclaimer
  • Privacy Policy
  • DMCA
  • Cookie Privacy Policy
  • Terms and Conditions
  • Contact us

Copyright © 2023 PouroverAI News.
PouroverAI News

No Result
View All Result
  • Home
  • AI Tech
  • Business
  • Blockchain
  • Data Science & ML
  • Cloud & Programming
  • Automation
  • Front-Tech
  • Marketing

Copyright © 2023 PouroverAI News.
PouroverAI News

Welcome Back!

Login to your account below

Forgotten Password? Sign Up

Create New Account!

Fill the forms bellow to register

All fields are required. Log In

Retrieve your password

Please enter your username or email address to reset your password.

Log In