In the run up to Microsoft Build, I spent time trying out the new features of Copilot. I had already used Copilot in (as well as Qwen, Claude, and ChatGPT) in VS Code for development. It was useful for some of the more boring aspects of programming such as formatting output and the "scaffolding" of code. Other than that, you get a lot of bad code and have to spend time debugging code you don't know. Understanding that, I thought I had reasonable expectations of Copilots low code app and task generator. Alas, I didn't realize that even my modest expectations were setting the bar too high.
Copilot Tasks and apps built around them is a great example of real value that AI could bring the average consumer or business. You tell Copilot about some activity that you want to automate, for example pulling information from the Internet into a daily digest, and any parameters around and "Voila!" it creates a web app to do it. While that sounds good in Microsoft's marketing literature, it doesn't really work that way.
The first problem is access to information. Now, if Copilot came back and said "without a connector or at least an RSS feed, we can't access the information" most of us would understand. This is, after all, technology not magic. Instead, Copilot goes along happily building an app for you only to have it not work. Ironically, sometimes the connectors to Microsoft's own products don't work, producing hilarious suggestions from Copilot. I tried to connect my Outlook account (I have a Microsoft Office 365 subscription) to Copilot to produce a digest of all my appointments and tasks for the day. It kept failing because Outlook insisted on 2FA for each query. Copilot being tagged a security risk by Microsoft's own products is funny enough, but it's suggestion that I switch to using Gmail instead was downright hilarious. I imagine someone in Microsoft marketing suddenly stopping whatever they were doing because they felt a profound disturbance in the force. It maybe it's the farce. Thankfully, Microsoft fixed that. Maybe it's what happens when you vibe code AI solutions.
Another strange problem is that Copilot doesn't seem to understand how to handle data even though the majority of their sample tasks and apps are data driven. In several cases, the data was pulled once and placed in an HTML CDATA block instead of using dynamic API calls. This meant that you had to rebuild the entire app or task in order to have it update. Copilot seemed to struggle with the idea of an app that updated daily not to mention on demand. Everything was hardcoded making it look right but completely static.
The Tale of Two Apps
I tried all kinds of apps and tasks as a test of what Copilot tasks and apps could due. I asked Copilot to create for tasks to send me daily digests of my calendar and tasks, create a web app to show me a daily list of calendar entries, tasks, and emails (data which comes out of one Microsoft application), retro skin a web site, and to create a list of all the baseball games for the day in both the major league and minor leagues with game times and where they can be viewed. Let me tell you about two of them.
Retro Me!
I love the website Hack A Day (hackaday.com). I've read this site for so long now that I can't remember when I began. It is likely about the time it started. I asked Copilot to skin it with a retro 80s BBS look. That would seem to be a fun homage to the site's maker content.
Now it's important to know that Copilot takes a long time to create these apps. A really long time. Sure, you can go off and do something else while it runs but 20 minutes is not unusual for it to complete. This means every iteration of the app takes time and you have to expect that it will take several iterations to get what you want.
After three iterations, I had what look like an awesomely skinned app. It correctly found the RSS feed and built around that rather than screen scraping the site. Copilot used the data to create something unique. It was great looking. Copilot used a black background with mostly green text and line characters to build something that could easily have been a BBS int he 80s.
Like I said, it was great... except that it didn't work. You would click on and menu entry, get a list of articles, click through to the article only to get an "Oops! That Page Can't Be Found" message. Copilot was creating links for articles that didn't exist. When I told Copilot that it didn't work, it would claim to understand why, rebuild the application, but nothing would change. It would give elaborate explanations but never could fix it.
I checked the Hack A Day RSS feed, the supposed data source, and the articles that Copilot was listing didn't exist int he fed. I searched the site and found there were no articles with those titles, let alone on those dates. Copilot just made up the data. When told this. Copilot did the equivalent of making excuses and kept trying.
Keep in mind, you don't see the code, like you would if you were using AI to vibe code. I can ask it for the code (and wait 20 minutes again) but there's no guarantee that it wouldn't break later.
The big problem these errors represent is obvious. This is supposed to be used by non-developers as is all low code/no code solutions. Theya re not going to be able to guide the AI to fix bugs. They will just get irritated and give up or (and this is much worse) go about their merry way blissfully ignorant that they are dealing with poorly functioning apps or worse, incorrect or made-up results.
Play Ball... Or Not.
I'm a huge baseball fan. I have an MLB TV subscription and watch a game each day, sometime two. The biggest hassle is looking through all the available games in the major and minor leagues, seeing what times they are on and how to view them. Some games are blacked out in my region, some are only on a streaming service I don't have (real annoying since I'm paying for this), and some are on broadcast TV which I can then watch. That's a lot to look through each day.
I asked Copilot to create a list of baseball games in the major and minor leagues, that were available that day, noting which were on MLBTV, regular TV, or some other streaming service. Other features were filters for different leagues and blacked out games. It needed to be updates daily but also on demand, since schedules change. If Copilot had come back and said that it didn't have access that information, I would have accepted that. It's not like I expect Copilot to have access to the APIs for several different league web sites.
Instead, Copilot happily went about building the app. Unfortunately, it built it stupid. It was able to retrieve the information from a set of APIs I didn't know existed but hardcoded the returned data into the app as CDATA blocks. That's not what "daily' or "refresh" means. With a structure like that, the app would have to be rebuilt each and every day (and there's that 20-minute time lag problem again). That's not exactly more efficient than just looking through the web sites myself.
This is problem number two with Copilot - it makes poor choices that no human programmer would make. Even a recent college grad knows to only use CDATA for static data. Dynamic data has to be reread when you need it to refresh. No matter how many times I explained this to Copilot, it just could grasp the necessity to reread the data from the APIs. No software development manager would keep a developer around if they had to constantly explain exactly how to fix a problem and have nothing change.
The AI Value Problem

