https://arxiv.org/abs/2304.13343

Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory SystemLarge-scale Language Models (LLMs) are constrained by their inability toprocess lengthy inputs. To address this limitation, we propose theSelf-Controlled Memory (SCM) system to unleash infinite-length input capacityfor large-scale language models. Our SCM system is composed of three keymodules: the language model agent, the memory stream, and the memorycontroller. The language model agent iteratively processes ultra-long inputsand stores all historical information in the memory stream. The memorycontroller provides the agent with both long-term memory (archived memory) andshort-term memory (flash memory) to generate precise and coherent responses.The controller determines which memories from archived memory should beactivated and how to incorporate them into the model input. Our SCM system canbe integrated with any LLMs to enable them to process ultra-long texts withoutany modification or fine-tuning. Experimental results show that our SCM systemenables LLMs, which are not optimized for multi-turn dialogue, to achievemulti-turn dialogue capabilities that are comparable to ChatGPT, and tooutperform ChatGPT in scenarios involving ultra-long document summarization orlong-term conversations. Additionally, we will supply a test set, which coverscommon long-text input scenarios, for evaluating the abilities of LLMs inprocessing long documents.~footnote{Working inprogress.}footnote{url{https://github.com/wbbeyourself/SCM4LLMs}}arxiv.org
올해 4월 26일, 틱톡 연구진들이 arxiv에 올린 논문

자체 제어 메모리 시스템으로 대규모 언어 모델을 위한 무한 길이 입력 용량 활용 이라는 제목의 논문

(대충 지들이 만든 SCM이라는 시스템으로 토큰을 무제한으로 늘릴 수 있다 어쩌구 저쩌구)


https://www.zdnet.com/google-amp/article/microsoft-tiktok-give-generative-ai-a-sort-of-memory/

Microsoft, TikTok give generative AI a sort of memoryThe programs use databases to beat ChatGPT at hundreds of turns of dialogue. The innovations can give any large language model ChatGPT-like abilities.www.zdnet.com

마이크로소프트도 저걸 기반으로 확장해서 적용 시도 중