Natural language in 2020 at-a-glance:
In natural language processing scale matters. More scale: as these models get bigger, they get better so fresh data comes in as more users ask more difficult questions. Data augmentation for NLP starts to catch up with where it is in computer vision. More specialized data, cleaner data will be a priority. Because so much text is memorization, overfit in the big models is a problem worth solving. Over-sampling of niche data for pre-training seems to be a tactic people use. Sub-reddits become sub-sub-sub reddits.Re-writing for style - as people deploy more transformer models in 2020, there may well be breakthroughs in this area because transformers are so good at detection of style and improv.More reinforcement learning - take lessons from computer vision (where individual pixels provide for backprop) and use RL to do the same in language.Transformers used on other AI use cases - initially, computer vision.Google, OpenAI, Facebook and Amazon all ...
- - - - -
The rest of this post is available for our Pro Members. Please login if you're a member. And, if you're not, subscribe now to get access to all of our Pro Member content!
Email us with any questions.