As technology continues to advance, we are rapidly approaching a future where Artificial General Intelligence (AGI) and AI glasses will become commonplace. While these advancements have the potential to revolutionize the way we live and work, they also pose a significant threat to our privacy. With the widespread use of AI glasses, everything we see and hear can be recorded and analyzed by powerful AI algorithms. Even seemingly insignificant details can be used to extrapolate a wealth of information about us, from our daily routines to our deepest desires. As AGI becomes more intelligent, Meta and other companies that control these technologies will have unprecedented access to our personal information. They will be able to track our every move, monitor our conversations, and even predict our behavior with alarming accuracy.
I for one embrace offline open source LLMs. IMO anyone using proprietary AI is a fool. Sure it may be a little better and easier to get started with, but I have never used it and never will. I can do stuff I never thought possible before. I can’t wait for the improvements that keep coming. If someone is only bright enough to use Windows, how can I fault them. Likewise with AI. If you want to whore absolutely every detail about yourself to these stalkerware companies that is your call. It is about ownership, and that is the real conversation we should all be having. A citizen at the very core is a person with a right to ownership. Proprietary is theft of ownership. Your personal data is a physical part of your person. Collecting and selling that data to attempt to manipulate you is an act of selling part of your person for exploitation. This has world changing long term implications that people are far too stupid to see. This is the turning point for a new age of feudalism. The AI is just a technology that can be used in many ways. It is not the problem. The theft of ownership and enslavement of your digital self is the key issue of our age. This is the end of citizenship and the beginning of the digital dark ages. It will set us back by hundreds of years of progress just like the last dark age when citizens became serfs because of greedy powerful feudal lords. In the next hundreds of years, people will only remember us as the peoples who willingly gave up their right to citizenship and ownership for free e-mail and searches on the internet that followed in a corruption cascade until “they owned nothing and they were happy about it” … until they learned what they really lost and could never get back until millions died to earn it once again.
What I can run with my hardware just isn’t comparable yet but I fully support the sentiment and plan to switch over once it gets a bit better.
A 70B at ~5bit quantization with GGUF streams at 1-1.5 tokens a second on a 12th gen i7 with a 16GBV 3080Ti and 64GB of system memory. I am running mostly on a gigabyte aorus laptop with those specs. If I could buy again, I would build a dedicated server tower and use a cheap laptop. I ended up using network hosted AI on my other devices a lot more than I expected. Right now, system memory is super important for the larger models. Machines with more CPU cores and at least 96GB of RAM are important. It is possible to use a swap partition on the storage drive. If you can hunt down a workstation with advanced AVX512 support (CPU ISA), that is probably the cheapest way to run really large models as quick as possible without enterprise GPUs and a $8k-$10k setup. I went from 4th gen i7 to 12th back in July. The difference is massive across the board. I would do it again.
I already have few RPis at home and a NAS so when I’ll change my desktop my “old” 32Go 2080ti Corsair will go to the basement, headless, and serve models using Dockers and HTTP endpoints, a continuation of https://fabien.benetou.fr/Content/SelfHostingArtificialIntelligence
Right now to be honest unless there is a desktop VR game that truly needs it, e.g a successor to Half-life: Alyx I’m in no rush. Kind of excited by the project though.