As technology continues to advance, we are rapidly approaching a future where Artificial General Intelligence (AGI) and AI glasses will become commonplace. While these advancements have the potential to revolutionize the way we live and work, they also pose a significant threat to our privacy. With the widespread use of AI glasses, everything we see and hear can be recorded and analyzed by powerful AI algorithms. Even seemingly insignificant details can be used to extrapolate a wealth of information about us, from our daily routines to our deepest desires. As AGI becomes more intelligent, Meta and other companies that control these technologies will have unprecedented access to our personal information. They will be able to track our every move, monitor our conversations, and even predict our behavior with alarming accuracy.
A 70B at ~5bit quantization with GGUF streams at 1-1.5 tokens a second on a 12th gen i7 with a 16GBV 3080Ti and 64GB of system memory. I am running mostly on a gigabyte aorus laptop with those specs. If I could buy again, I would build a dedicated server tower and use a cheap laptop. I ended up using network hosted AI on my other devices a lot more than I expected. Right now, system memory is super important for the larger models. Machines with more CPU cores and at least 96GB of RAM are important. It is possible to use a swap partition on the storage drive. If you can hunt down a workstation with advanced AVX512 support (CPU ISA), that is probably the cheapest way to run really large models as quick as possible without enterprise GPUs and a $8k-$10k setup. I went from 4th gen i7 to 12th back in July. The difference is massive across the board. I would do it again.
I already have few RPis at home and a NAS so when I’ll change my desktop my “old” 32Go 2080ti Corsair will go to the basement, headless, and serve models using Dockers and HTTP endpoints, a continuation of https://fabien.benetou.fr/Content/SelfHostingArtificialIntelligence
Right now to be honest unless there is a desktop VR game that truly needs it, e.g a successor to Half-life: Alyx I’m in no rush. Kind of excited by the project though.