AI in the real world — 3. Make your own Grammarly
Grammarly is one of the major breakthroughs that AI has made in this decade. I am presuming that most of us know what Grammarly is and how we are hugely benefited by such a tool. What if I tell you that you can make your own minimal spell corrector tool easily without expertise in coding using Transformers. Well, let’s not wait and dive into the post.
To my understanding, the complimentary version of Grammarly does these three major tasks:
- Look for spelling errors
- Paraphrase the sentences
- Provides us with the emotion of the sentences
So we will not be able to achieve performance similar to Grammarly but we can easily reach base accuracy by building a simple sequence to sequence model for spelling errors and zero-shot learner for the classification task.
Prerequisites
- A system with at least 2GB of GPU support
- Transformers
- Torch
- Around 50k random sentences of data
Modelling
For building any model we need data, if you can’t get enough data fear not, use any text generation models like GPT or even any text to text generation models that would work to generate around 50k to 100k sentences. Then use this simple piece of code to randomly impute spelling errors in the sentences so that now you’ll have a source as corrupted sentences and target as grammatically correct sentences.
The above code snippet would impute errors in randomly selected words of a sentence. The parameter n_words is how we can control how many words in a sentence do we need to impute the errors on.
I have taken T5 small and used around 50k sentences as the dataset for building the model. The main reasons for choosing a T5 model are because they are one of the best in seq-seq modelling tasks, and due to the attention mechanism in the transformers, they outclass LSTMs or RNNs with regards to efficiency and performance. For implementing the modelling part you can look into the official tutorial of transformers. The mentioned notebook explains fine-tuning a t5-small model for summarization task, but by changing the prefix for each sentence you can fine-tune it for a different task as your wish. If you want to skip all these and need a simple wrapper with configurable parameters feel free to try out this. This is basically a simple wrapper that I wrote for fine-tuning NLP models currently, it supports only these three:
- bert-base-uncased for classification tasks
- t5-small for seq-seq tasks.
Once you have fine-tuned your model you try out yours or use the below code snippet to leverage the one which I have already fine-tuned on.
The above code instantiates two transformers models respectively for the spell correction and classification of emotion in the sentence given. For the latter kindly go through the documentation given by transformers on zero-shot classification. Please find below a simple example of how the above code works
input_text = input()
gram = Grammarly(input_text)
gram.grammarly()Output:
input_text = I am happu to inform that w ehave won the competetio{'emotion': 'happy',
'emotion_score': 0.5591591596603394,
'modified_text': 'I am happy to inform that i have won the competition'}
input_text = That is sad to har the passing away of hs grandfather{'emotion': 'sad',
'emotion_score': 0.5666968822479248,
'modified_text': 'That is sad to have the passing away of his grandfather'}
Well, we would not be completely satisfied with what we have seen above but at least there is a significant part where the spell corrector was able to rightly get the word that is incorrect and replaces it with words that would be semantically near to the context of the sentence. By using a bigger model i.e (t5-large or even T0 from BigScience which beat the GPT-3 from OpenAI for many tasks) or increasing the training data, tuning the parameters we can achieve better results. I fine-tuned the model with a max sequence length of 32 and the model used here would not work great on longer sentences. That said for smaller and simpler sentences this model would work as great as ever
Okay let’s wrap up as I feel like dragging on X), thank you all for your valuable time and support. Bye and take care until next time.
