Great Test for Project Nomad

Lately, I have been implementing my own set of computing services. Some of you might recall my NextCloud project to implement my own set of cloud services. I did that primarily to reduce my reliance on the cloud services of a certain company in Redmon, WA. It's not that I think that are evil or, at least, no more evil than any other tech company. It's just that all of these cloud service companies have a history of dropping and changing services that I've come to rely on. It was also good geeky fun.

In that vein, I've just completed a new project to implement information services as self-hosted services. The idea behind self-hosted information services is to provide access to important information when the Internet is unavailable. I could potentially turn off my Internet service or go out into some place off-the-grid and still have essential information. 

Now there are three classes of people who do this:

1. Techie geeks me who do it because we can
2. Folks who live somewhere that internet service is unreliable or unavailable
3. Preppers

Many of these systems deliver internet-based information to schools in places that have no or poor connectivity. The Preppers are something else altogether. These are folks who want to be prepared (prepped) for disaster. There is a bit of overlap with survivalists but, as far as I can tell, preppers are more about readiness then wanting to hide from humanity or the government. For the record, I'm in the first category.

There were a number of options for software, but I chose to use Project Nomad. Project Nomad is an open source control plane for implementing and managing other services such as Kiwix. It runs on Linux (because Docker) and I was able to use an old laptop that I had already loaded Ubuntu onto. Project Nomad is one of the easiest bits of server software I've ever installed. It sets up Docker containers, installing Docker, if necessary, asks what you want to install, and then just does it. Once the core software is in place, you can choose what content to download which happen in the background.

There are five major components, and you can choose any or all of them to install. They are:

1. An information library, including Wikipedia. powered by Kiwix
2. Maps from OpenStreetMaps
3. A drug database
4. A learning platform with content from Khan Academy
5. An AI LLM

I went with the information library, maps, and the drug database. The AI component was not going to run well on the hardware I had, and I really didn't care if I had AI support. The learning platform was just not something I'd ever really use. It would be different if I was trying to run a school in a caven. Any service can be removed or added later as well as a few other services such as Jellyfin media server and Vaultwarden password manager. 

The next step is telling Project Nomad what content you wanted to download. I choose Wikipedia, Wiktionary, some Stack Exchange forums on technical subjects, a bunch of categories of books from Project Gutenberg, a collection of medical information including military medical books, the iFixIt website, and a bunch of prepper material. That last bit of information was mainly out of curiosity. There was an entire set of videos on how to put together a bug out bag. It's not what I normally think about.

A lot of the information is useful to have around, internet or not. The real test came when I experienced a substantial ISP outage. Project Nomad meant that I had a lot of useful information still available and I wasn't floundering. It worked perfectly with no glitches or hidden connections to internet sites. The only thing Project Nomad couldn't do was pull updates for the software and content. That's to be expected. 

That need to pull information down, at least initially, is the Achilles Heel of Project Nomad. You need a stable internet connection to install the software and content. If you have your Project Nomad server already set up before the zombie apocalypse, then you're fine. If you didn't think about it until the manure hit the rotating blades then you're out of luck.

At the moment, Project Nomad is mild assurance against sporadic internet outages. It could also be setup by someone and shipped to someone else off-the-grid. It's easy to install and pretty low on resource usage, if you skip the AI. That's fine because when the robot apocalypse happens, I'll be trying to kill AI not run it off soar energy in my cave.

1. 
   

Microsoft is Not My Copilot

In the run up to Microsoft Build, I spent time trying out the new features of Copilot. I had already used Copilot in (as well as Qwen, Claude, and ChatGPT) in VS Code for development. It was useful for some of the more boring aspects of programming such as formatting output and the "scaffolding" of code. Other than that, you get a lot of bad code and have to spend time debugging code you don't know. Understanding that, I thought I had reasonable expectations of Copilots low code app and task generator. Alas, I didn't realize that even my modest expectations were setting the bar too high. 
Copilot Tasks and apps built around them is a great example of real value that AI could bring the average consumer or business. You tell Copilot about some activity that you want to automate, for example pulling information from the Internet into a daily digest, and any parameters around and "Voila!" it creates a web app to do it. While that sounds good in Microsoft's marketing literature, it doesn't really work that way.
The first problem is access to information. Now, if Copilot came back and said "without a connector or at least an RSS feed, we can't access the information" most of us would understand. This is, after all, technology not magic. Instead, Copilot goes along happily building an app for you only to have it not work. Ironically, sometimes the connectors to Microsoft's own products don't work, producing hilarious suggestions from Copilot. I tried to connect my Outlook account (I have a Microsoft Office 365 subscription) to Copilot to produce a digest of all my appointments and tasks for the day. It kept failing because Outlook insisted on 2FA for each query. Copilot being tagged a security risk by Microsoft's own products is funny enough, but it's suggestion that I switch to using Gmail instead was downright hilarious. I imagine someone in Microsoft marketing suddenly stopping whatever they were doing because they felt a profound disturbance in the force. It maybe it's the farce. Thankfully, Microsoft fixed that. Maybe it's what happens when you vibe code AI solutions.
Another strange problem is that Copilot doesn't seem to understand how to handle data even though the majority of their sample tasks and apps are data driven. In several cases, the data was pulled once and placed in an HTML CDATA block instead of using dynamic API calls. This meant that you had to rebuild the entire app or task in order to have it update. Copilot seemed to struggle with the idea of an app that updated daily not to mention on demand. Everything was hardcoded making it look right but completely static.

The Tale of Two Apps

I tried all kinds of apps and tasks as a test of what Copilot tasks and apps could due. I asked Copilot to create for tasks to send me daily digests of my calendar and tasks, create a web app to show me a daily list of calendar entries, tasks, and emails (data which comes out of one Microsoft application), retro skin a web site, and to create a list of all the baseball games for the day in both the major league and minor leagues with game times and where they can be viewed. Let me tell you about two of them.

Retro Me!

I love the website Hack A Day (hackaday.com). I've read this site for so long now that I can't remember when I began. It is likely about the time it started. I asked Copilot to skin it with a retro 80s BBS look. That would seem to be a fun homage to the site's maker content. 

Now it's important to know that Copilot takes a long time to create these apps. A really long time. Sure, you can go off and do something else while it runs but 20 minutes is not unusual for it to complete. This means every iteration of the app takes time and you have to expect that it will take several iterations to get what you want. 

After three iterations, I had what look like an awesomely skinned app. It correctly found the RSS feed and built around that rather than screen scraping the site. Copilot used the data to create something unique. It was great looking. Copilot used a black background with mostly green text and line characters to build something that could easily have been a BBS int he 80s. 

Like I said, it was great... except that it didn't work. You would click on and menu entry, get a list of articles, click through to the article only to get an "Oops! That Page Can't Be Found" message. Copilot was creating links for articles that didn't exist. When I told Copilot that it didn't work, it would claim to understand why, rebuild the application, but nothing would change. It would give elaborate explanations but never could fix it.

I checked the Hack A Day RSS feed, the supposed data source, and the articles that Copilot was listing didn't exist int he fed. I searched the site and found there were no articles with those titles, let alone on those dates. Copilot just made up the data. When told this. Copilot did the equivalent of making excuses and kept trying.

Keep in mind, you don't see the code, like you would if you were using AI to vibe code. I can ask it for the code (and wait 20 minutes again) but there's no guarantee that it wouldn't break later.

The big problem these errors represent is obvious. This is supposed to be used by non-developers as is all low code/no code solutions. Theya re not going to be able to guide the AI to fix bugs. They will just get irritated and give up or (and this is much worse) go about their merry way blissfully ignorant that they are dealing with poorly functioning apps or worse, incorrect or made-up results. 

Play Ball... Or Not. 

I'm a huge baseball fan. I have an MLB TV subscription and watch a game each day, sometime two. The biggest hassle is looking through all the available games in the major and minor leagues, seeing what times they are on and how to view them. Some games are blacked out in my region, some are only on a streaming service I don't have (real annoying since I'm paying for this), and some are on broadcast TV which I can then watch. That's a lot to look through each day.

I asked Copilot to create a list of baseball games in the major and minor leagues, that were available that day, noting which were on MLBTV, regular TV, or some other streaming service. Other features were filters for different leagues and blacked out games. It needed to be updates daily but also on demand, since schedules change. If Copilot had come back and said that it didn't have access that information, I would have accepted that. It's not like I expect Copilot to have access to the APIs for several different league web sites. 

Instead, Copilot happily went about building the app. Unfortunately, it built it stupid. It was able to retrieve the information from a set of APIs I didn't know existed but hardcoded the returned data into the app as CDATA blocks. That's not what "daily' or "refresh" means. With a structure like that, the app would have to be rebuilt each and every day (and there's that 20-minute time lag problem again). That's not exactly more efficient than just looking through the web sites myself. 

This is problem number two with Copilot - it makes poor choices that no human programmer would make. Even a recent college grad knows to only use CDATA for static data. Dynamic data has to be reread when you need it to refresh. No matter how many times I explained this to Copilot, it just could grasp the necessity to reread the data from the APIs. No software development manager would keep a developer around if they had to constantly explain exactly how to fix a problem and have nothing change. 

The AI Value Problem

This just outlines the main problem with AI adoption, especially with business. No one is going to spend money on something that doesn't return good value for that money. That's true with personnel and technology alike. If all anyone gets from AI is chatty searching, document summarization or intelligent email responses, then the ROI for AI is allusive, maybe non-existent. The ability to generate attractive apps that can act as business dashboards or consolidate information by just describing what is desired has a lot of value. Even current no code solutions still require work and a knowledge of the data sources that most folks don't want to get involved with. To have an AI product crank out personalized views of data is very valuable indeed, but only if it works right. Making up data, especially, makes it worse than useless. 

To be fair, Copilot Tasks and Apps are still in preview. Some bugs are expected. These issues, however, are fundamental. They are not minor formatting issues; they are showstoppers and time wasters. Let me say it this way: I have not been able to get a correctly functioning application out of Copilot Tasks and Apps. 

Conceptually, what Microsoft is doing makes sense. The devil, as the old saying goes, is in the details. If you can't get data handling right, can't access your own products, and make stupid programming choices that the most inexperienced developer wouldn't make then you have a demo or proof of concept, not a product remotely ready for general release or even a preview. 

I can't help but wonder how long consumers and business will put up with half backed AI while waiting for the promises to come true.

So, I Built a Cloud

What's that you say?

That's right. I built myself a cloud in my home lab. I used old computers and off-the-shelf free and open-source software (FOSS) to create a system that approximates the functionality of Google Workspace. 

Two things I immediately learned from this project. First, like a Tiki bar (IYKYK) it's never really done. There's always tweaks and new features to add. Second, you have to attend to it like any complex system, especially patches and updates.

Why did I build a Cloud?

The fact that I built a cloud system should immediately raise the question "Why?" There were three reasons really. First and foremost, I have concerns about being tethered to big tech for all of my computing needs. I have been using one form or another of the Microsoft technology stack as my daily drive for the past 40 years, starting with MS-DOS and Word back in the 80s. The difference between then and now is that I wasn't as reliant on Microsoft for so many applications. I could easily have swapped out my word processor since no one transmitted electronic files in those days. As time went on, however, more and more of my computing was being provided by Microsoft so that now I rely on them for my OS, all of my office applications, all of my communication applications, and a big chunk of my creative apps. Woof! Worse yet, I heavily rely on OneDrive for storage and the ability to access files remotely. If Microsoft decided to change something I didn't like, there would be little I could do about that.

A case in point is the discontinuation of Microsoft Publisher. Publisher has always been a desktop resident application; There is no online Publisher. Yet, when Publisher goes extinct, as it will this year, even the desktop version on my computer will stop working. In the bygone days, if a software developer stopped working on a product, what you already owned would continue to work albeit without updates. In the age of rented software, that's no longer true. When a company sunsets a cloud or subscription application, it disappears or stops working. There are entire websites devoted to the graveyard of big tech products because it happens so often. Having my own cloud makes me less vulnerable to big tech's tendency to cut bait and leave customer dangling.

Another reason for building my own cloud was so that I could reuse a bunch of hardware I had sitting around. Frankly, I hate to throw out usable stuff, especially when much of what's in that stuff is toxic to the environment. I live by the maxim "Reuse, Repurpose, Recycle!" I had recently gone through an upgrade cycle of my household's main desktop and laptop computers and had a bunch of hardware sitting around idle. most of it was under 10 years old. Add to that a bunch of ancient (16 years and 20+ years old respectively) hardware that was cluttering up my home lab, and it seemed like a good time to put it to use.

Finally, it seemed like a fun project. I know, not everyone would hear "building a cloud service" and think "Hey that's fun!" What can I say, I'm a computer geek. So, with that in mind, we were off to the races.

The Stack

I spend too many years in product development not to have a base set of requirements. In this case, hey didn't need to be detailed or stringent. I wanted to allow the project to evolve. Still, I needed a starting point. With my goals in mind, this is what I came up with:

1. It must be FOSS. If my primary goal was freedom from big tech, relying on anything but FOSS was a mistake. Now, that meant relying on middle sized tech. For example, I knew that it would be a Linux stack. I choose Ubuntu Linux because I had the most experience with it. I could have chosen a project that was not tied to some company but that would have increased the degree of difficulty. Sorry to say, the best Linux implementations are from corporations not hobbyists.
2. It must approximate Google Workspace. To that end, the software I used had to have enough functionality that was either core or from a community that could support remote file access, online office applications, email and calendar, preferably a bookmark manager, and a bunch of other smaller features such as notes. I deemphasized social media functionality and video conferencing since this was mainly for my use and I knew I didn't have the hardware to really run video well. 
3. It has to run on more limited hardware. If I had to buy a rack of new servers to run the software, then it would defeat the purpose of reusing the old hardware. This was another reason to use Linux since you can always find a distribution that will run on more limited hardware.
4. It did not need to be multi-tenant, only multi-user. Google, Microsoft, Zoho, Proton et al have to run their multiple instances on shared hardware platforms. That's why they deploy multi-tenant systems. For my cloud, I only had to support multiple users in the same implementation, not multiple instances. This vastly simplified my requirements. I didn't need something like Cloud Foundry or even Kubernetes to run separate instances of my cloud software.
5. It had to all be off the shelf. Outside of a few maintenance scripts, I didn't want to have to develop my own software. I may in the future extend my cloud to include a set of services that don't yet exist but that's an entirely different project.

In the end, I came up with the following software stack:

• Ubuntu Linux. It's stable, supported, and I had a lot of experience with it. I opted for the desktop versions so that I could make use of the GUI tools. If this was an enterprise stack instead of a home lab, it would be different. There I would likely not have a desktop absorbing resources and instead have Kubernetes running a bunch of containers. 
• Snap containers. I know this could be controversial but snap containers are built into Ubuntu already and don't need any additional tooling to run. Meanwhile, they approximate the sandbox capabilities of other container systems such as Docker. I've used Docker before and found, for my more limited purposes, snaps and Docker containers are about the same.
• SAMBA and SFTP. Most of the time, I need to access my files from my internal Windows environment. SAMBA provides the SMB remote file capabilities that are most recognizable to Windows and can work with Linux as well. Using SAMBA meant that I could sync OneDrive to my file store without too much effort. SFTP allowed for network access to files from within containers, which turned out to be a good thing. I didn't want to have to implement something like a service mesh to get secure file access across networked devices.
• OpenSSH. Secure connections are a thing. I use SSH to remote into my servers and implement other secure protocols such as SFTP. That's expected.
• Nextcloud. This was the meat of the software stack. Nextcloud is pretty well known and implements all the cloud basics. There is a vibrant community that produces other applications for Nextcloud and is available as snap and Docker containers, as well as standard Debian packages. There are also a series of mobile apps for Android that allow access to most major functions. 

Physical Architecture

Like I said, I had a bunch of leftover hardware floating around my home lab. Besides sitting idle, it sort of irritated my wife that all of this "useless" stuff was cluttering up my office. Useless? Ha!  

As a side note, using leftover hardware created some challenges. I wanted to add another hard drive (yes, I have a bunch of those lying around too) to one of the computers only to discover that all the SATA power connectors on the motherboard were in use. There were solutions for that problem, but it was not something I wanted to get into, especially since it meant buying more stuff. 

Anyway, I decided to separate the file store from the application server and add a bunch of other stuff to make it work better. This included network components, extra external archive storage, keyboards, monitors and network equipment. And yes, it was all leftovers.

The physical architecture ended up in three parts like this:

• A file server. This contained a 1T disk, 16G of RAM, and 8 core processor. I had to place this in an out of the way spot in my house because it has a new but loud fan. Rather than connect it via the onboard WiFi, I used an extra remote station for my mesh network. I was able to attach the file server to the ethernet port on the remote station and achievemuch better performance than WiFi to a remote or base station. As I said, I used a desktop version of Ubuntu so that I could use the GUI tools, so this also needed a monitor, keyboard, and mouse.
  The file server implemented SAMBA which made it accessible to Windows and Linux clients as file shares. This made it was immediately useful to my Windows clients. It also implemented an SFTP server to facilitate network connections from snap containers.
• An application server, or app server. The app server has a 500G hard drive, 16 core i7 processor, and 8G of RAM. The app server is wired into a switch that connects directly to the router. Its traffic has priority on the switch since the other devices on the switch are clients of some sort. Attached to the app server, besides the usual array of keyboard, monitor, and mouse, is a 1T USB drive. It is available on the network via SAMBA but is quite slow. That's fine since it only mirrors the most important files on the file server. It acts as a backup in case the file server is offline. 
  The app server runs the Nextcloud software in a snap. You can look up Nextcloud to see it's full range of capabilities but needless to say, it does the job well. Nextcloud rarely pushes any one core into double digit utilization, everything including the OS takes up less than half the disk space, and memory is typically only at about at 30% utilization. Needless to say, there is a lot of headroom on this box for more services.
• A monitoring station. I could monitor everything from my primary desktop, but it has other things to do, like running some AI crap. Instead, I have a 16-year-old Dell laptop that runs Pop-OS (which is built on Ubuntu) in use for monitoring the boxes. It only has 4G of memory and a 128G SSD which makes it hard to use as a real laptop. It can operate as something akin to a Chromebook but even then, it's WiFi is slow and a two-core processor. It's just too slow to be truly useful. As a station for monitoring a couple of boxes, it's great. One more piece of equipment saved from a landfill. 
  I put Pop-OS on it because... well, why not? I wanted to see if the Cosmic desktop was all that folks were saying it was. Originally Pop-OS, which is developed by System7, used a heavily modified version of Gnome to achieve its signature look and feel. It was more of a mod than a new desktop. With its last release, System7 implemented the new Cosmic desktop written entirely in Rust. It uses all the familiar desktop elements such as a toolbar and popup launcher, along with a quick command line launcher, similar to PowerToys Command Palette. It looks pretty and is very responsive. It's also a bit unstable and I've already had to reinstall Cosmic when it became corrupted. But I digress.
• 
  

No-IP makes it really cloudy 

A cloud system that is only available within your interior network is not very cloudy at all. If it can't be accessed from the wider world, then you can't unshackle yourself from the bonds of big tech. I'm a cheapskate so I didn't want to pay a lot of money to my already annoying and expensive ISP to get a static IP address. Thankfully there is a good solution. It's called Dynamic DNS or DDNS. 

See, the problem of accessing a cloud service from an everchanging IP address is that you can't know from moment to moment what that address might be. Subsequently, the internet naming system that translates your numerical address into something a human can remember can't know what your IP address is if it keeps changing. DDNS detects changes in the IP address from your ISP and updates DNS records with that new address. 

No-IP is a service provider for DDNS and a bunch of other network services. They have a free tier DDNS that is perfect for home labs. My router already supports No-IP so I didn't need to do anything fancy on my app server to make it work. No-IP provides me with a human readable address for my network and keeps it's IP address updated as my router detects them. 

A few changes in my firewall and I now have a cloud system that is available from web browsers and Android apps throughout the world. I had the opportunity to try it out from Europe, and it worked perfectly, even with the natural latency that comes with sending packets halfway across the world. 

Final results

I have a cloud! More accurately, I have a set of web and mobile services that are accessible from anywhere I have an internet connection. These services provide pretty much the same set of services I would use from Google, Zoho, or Microsoft, give or take a few. Nextcloud is not perfect and I can see some instances where I might need to develop an app our two. 

I haven't abandoned the services I use from Microsoft, or Google or Zoho but I have a Plan B in case any of them raises prices a lot or cut back on services dramatically. My daily drive still comes from big tech, but I am no longer completely reliant on them. That, and I have a bunch of computer hardware that didn't end up in a landfill, and I had some fun doing it. 

It feels good to know that I didn't add to pollution, still have some technical chops, and can give a middle finger to big tech whenever I want. Nice!