Can Devin ai actually replace web developers?

There’s a lot of hype around Devin ai.

And developers are freaking out.

In this example, Site Reliability Engineer, Mervin Praison, gives Devin the task of scraping his website, extracting all the information, putting each page in a markdown file, and zipping all the files in one folder so he can download it.

Devin details its plan in the code editor:

And then it carries out the plan.

On the first try, it was pretty good. Devin scraped some pages and extracted all the key information:

However, not all of the website’s pages were scraped (6 out of 76), so Mervin prompted Devin to do a deep crawl. He wanted all of the pages to be captured:

For a simple task like this, Devin does a stellar job. It just needs a little hand holding.

But what about more complex tasks?

Devin isn’t there just yet…

Let’s look at the demo videos.

In “Debunking Devin”, software engineer with 35 years of experience, “Internet of Bugs”, digs into the claim that Devin can complete jobs on Upwork.

Here’s what he finds.

Point #1: The Devin team cherry picked “road damage” jobs.

This in itself isn’t damning. 

However, it does suggest this Upwork job wasn’t chosen randomly:

Point #2: “Internet of Bugs” spells out what a developer would need to get from the customer in order to complete the job.

Of course, Devin doesn’t ask for any of this stuff:

“This is the part of the job of a software developer that AIs are bad at. The important part, the difficult part, the time-consuming part of being a software engineer is communication with your customer/stakeholders. Figuring out what actually needs to be done, going back and forth, saying, ‘Okay, this would be a lot easier, let’s do that’. Those are the things that AI just isn’t capable of doing […]” – “Internet of Bugs”

As such, the deliverables for this task should contain:

• Which Cloud instance type to use
• Which Cloud Operating System Image to use
• How to setup install environment (CUDA, Apex, PyTorch, etc)
• How to install roadDamageDetection2020
• How to update for modern python et. al. or how to install 4-year-old environment
• How to get data you want to use onto the instance
• How to get the output off the instance

Point #3: Devin seems to be creating files with errors in them and then fixing the errors. This isn’t implied what Devin is doing.

The video gets pretty technical at this point, so it’s worth a watch.

Point #4: It takes Devin a long time to complete the task and it does a bunch of unnecessary stuff.

“All I had to do to replicate Devin’s results was get an environment set up on a cloud instance with the right hardware and run literally two commands with the right paths.

“All of this stuff makes it look like Devin did a lot of work […] All of those code fixes aren’t relevant at all because it’s all code that Devin generated itself.”  – “Internet of Bugs”

All of this is to say that Devin has flaws, and the demo videos (particularly the Upwork one) overstate what Devin can do.

For the record, “Internet of Bugs” says, “Devin can do some impressive things. I just wish the company had been truthful and took the win – but they didn’t.”

This is just one example, but it highlights an important point:

There’s a lot of hype around AI.

So before you blindly agree with anything you read on the internet, do your diligence and check the facts.

This isn’t to say Devin won’t change and continue to improve. You can bet your bottom dollar it will. But right now, Devin won’t be replacing web developers anytime soon.

Not unless you want a bunch of mumbo-jumbo in your codebase.

Meet available, vetted talent today

Looking to hire actual human developers who will do what you want them to do?

We can help!
If you’re looking for the very best tech talent, contact us today to see what we can do for you.