AI’s natural language frontier – context is king but just how big will models get?

Sign up for our newsletter, Artificiality, to get our latest AI insights delivered directly to your email inbox.

Natural language in 2020 at-a-glance:

In natural language processing scale matters. More scale: as these models get bigger, they get better so fresh data comes in as more users ask more difficult questions. Data augmentation for NLP starts to catch up with where it is in computer vision. More specialized data, cleaner data will be a priority. Because so much text is memorization, overfit in the big models is a problem worth solving. Over-sampling of niche data for pre-training seems to be a tactic people use. Sub-reddits become sub-sub-sub reddits.Re-writing for style - as people deploy more transformer models in 2020, there may well be breakthroughs in this area because transformers are so good at detection of style and improv.More reinforcement learning - take lessons from computer vision (where individual pixels provide for backprop) and use RL to do the same in language.Transformers used on other AI use cases - initially, computer vision.Google, OpenAI, Facebook and Amazon all ...

- - - - -


The rest of this post is available for our Pro Members. Please login if you're a member. And, if you're not, subscribe now to get access to all of our Pro Member content!

Subscribe here.

Email us with any questions.



At Sonder Scheme, we are on a mission to help humans win in the age of AI. We have created a human-centered AI design system that allows leading companies to create AI in an inclusive way. Companies like The New York Times, Starbucks, R/GA, Galp and the National Headstart Association tap us for engaging presentations at team/executive forums, in-depth workshops or long-term learning journeys. We support them with executive coaching and online access to our design system and up-to-the-minute insights into the frontiers of AI technology and ethics.

Share on email
Share on facebook
Share on linkedin
Share on twitter
Scroll to Top