A developer named Scott Cate at Microsoft has built a bot that does image recognition, and, apparently Ed Sheeran recognition as well.
Like any bot worth it’s salt, this one runs on Twitter and can be summoned using the #vision_api hashtag. After a slight delay (presumably because a computer in a cloud somewhere is examining the photo with a monocle), a reply is sent with a verbal description of the image.
#vision_api pic.twitter.com/nblqbhI7NA
— Scott Cate, Seattle #MSBuild (@scottcate) April 3, 2018
Confidence: 91.07 %
— #vision_api (@vision_api) April 3, 2018
Vision API: a sofa in a room
Full API Result: https://t.co/BzV6Jh3BUi
The bot is pretty accurate as far as artificial intelligence goes. That monocle is doing it’s job. Most real-world objects seem to be readily identifiable. And of course it wouldn’t be a proper technological achievement without being able to correctly distinguish hot dogs from…..everything else.
Another use of #vision_api pic.twitter.com/wmPMDFW6lD
— Scott Cate, Seattle #MSBuild (@scottcate) April 4, 2018
Confidence: 96.11 %
— #vision_api (@vision_api) April 4, 2018
Caption: a hot dog on a bun
Tags: hot, dog, food, dish, sandwich, bun, hotdog, paper, fries, sitting, bread, plate, long, table, covered
Full API Result: https://t.co/0GVsvX0Fjy
People have been tagging their photos on Twitter for the bot, and the internet does not disappoint. If you’re going to be a bot on Twitter, you can’t half-ass it.
#vision_api pic.twitter.com/nV2f42glgP
— Gabriel Robert (@kvlt_grobert) March 30, 2018
Confidence: 50.75 %
— #vision_api (@vision_api) March 30, 2018
Vision API: a group of toy people
Full API Result: https://t.co/G3B4VKNEvw
Not quite, bot, but good guess. I suppose “Cow Onesie” is a tad weird. What will the humans do next!?!
Hey @vision_api what is this?
— René Schulte @ #MSBuild (@rschu) April 2, 2018
My 5 year old said it's an Easter bunny dog. 🤣 pic.twitter.com/VdO7qZL0Zs
Confidence: 24.62 %
— #vision_api (@vision_api) April 2, 2018
Vision API: a close up of a bird
Full API Result: https://t.co/8ASKa051PU
Uh, no. But to be fair, I have no idea what that is either. Is that supposed to be a rabbit? That is honestly terrifying.
I happen to notice the trend and tagged a picture of a cheeky Ed Sheeran. I figured the bot would know that it was a man, but I did not expect that the bot would know WHICH man.
mom said you had to let me use the xbox pic.twitter.com/jwBf2AEf1Q
— dallas (@mixedhunty) April 1, 2018
Confidence: 86.74 %
— #vision_api (@vision_api) April 4, 2018
Vision API: Ed Sheeran standing in a room
Full API Result: https://t.co/cjZrR3WvWv
THAT’S ED SHEERAN. Even the machines know. You can’t be artificially intelligent if you can’t identify the talent who once crooned out, "And I know you love Shrek / Because we've watched it 12 times."
OMG! ED SHEERAN, AM I RIGHT? 😍 pic.twitter.com/M7D6FyDqYs
— Bëłłä Κ Łø:śt (RT PINNED) (@Azzurioak) April 10, 2018
You are SO right!
Confidence: 93.70 %
— #vision_api (@vision_api) April 11, 2018
Caption: Ed Sheeran wearing a blue shirt
Tags: person, man, clothing, shirt, looking, sweater, holding, front, wearing, standing, hand, black, blue, using, young, phone
Full API Result: https://t.co/tQX3medXjX
Not to mention he’s “wearing, standing” and “holding”. You look great no matter what you do, Ed!
As it turns out, the bot is not mutually exclusive to Ed. It can recognize most celebrities. Even the ones that you probably wish it wouldn’t.
Martin Shkreli investor to recover losses https://t.co/ZRTK5O4yQs pic.twitter.com/4KyPg4m1HT
— Former SEC Branch Chief (@LisaBraganca) April 11, 2018
Confidence: 84.31 %
— #vision_api (@vision_api) April 11, 2018
Caption: a man standing in front of Martin Shkreli, Benjamin Brafman posing for the camera
Tags: person, man, suit, standing, outdoor, posing, holding, photo
Full API Result: https://t.co/R7gB77N04M
Although sometimes it does get confused, like when it mistook Scarlett Johansson for Elizabeth Olson in this Avengers poster.
Testing #vision_api with an #Avengers #InfinityWar poster pic.twitter.com/5u57N1g6FL
— Shahed Chowdhuri @ Microsoft (@shahedc) March 29, 2018
Confidence: 31.95 %
— #vision_api (@vision_api) March 29, 2018
Vision API: Elizabeth Olsen, Danai Gurira, Chris Hemsworth are posing for a picture
Full API Result: https://t.co/kgTEehykUT
Apparently Chris Hemsworth is unmistakable, even with one eye missing. But Elizabeth Olson For Scar Jo? Come on. They aren’t exactly identica……HOLY CRAP ELIZABETH OLSON IS IN THE PICTURE…
It can get things pretty wrong too. It seems to have trouble with identifying “spikey bracelets”…
#vision_api pic.twitter.com/MQFdrqA5cp
— Purple Puppy🐾 BLACK CAT D.VA (@PuppyOwO) April 7, 2018
Confidence: 48.77 %
— #vision_api (@vision_api) April 7, 2018
Maybe: a person sitting on a bed
Tags: person, indoor, sitting, man, looking, small, top, dog, table, bed, little, young, holding, woman, red, white
Full API Result: https://t.co/LkIqrAQlld
Maybe. Or maybe just someone who needs to clean their room, but wants to look really good doing it.
And it really has a problem with furries. Not an ideological problem, but definitely a problem.
･ ｡
— SwiftOnSecurity (@SwiftOnSecurity) April 7, 2018
☆∴｡
･ﾟ｡★･
･ ﾟ｡ *
･ ﾟ｡･ﾟ★｡
☆ﾟ･｡°. ﾟ
ﾟ｡·･｡ ﾟ
ﾟ .｡☆｡★ ･
☆ ｡･ﾟ*.｡
* ★ ﾟ･｡ * ｡
･ ﾟ☆ ｡ pic.twitter.com/i9YUwZWFeK
Confidence: 46.07 %
— #vision_api (@vision_api) April 7, 2018
Maybe: a dog sitting on a bicycle
Tags: bicycle, dog, sitting, black, cat, white, standing, brown, large, man
Full API Result: https://t.co/igqN0xGoSr
This is all made possible by Microsoft’s Computer Vision API, which says that it can “Extract rich information from images to categorize and process visual data.” Aside from being able to recognize celebrities and pharma bro’s, it can also recognize handwriting in images and popular landmarks as well. Microsoft says that it can recognize o”200K celebrities from business, politics, sports and entertainment.” as well as “9000 natural and man-made landmarks”.
You can try out the bot yourself by adding the #vision_api hashtag to any tweet with an image. Now go and have fun seeing if the bot loves Shrek as much as Ed does.
I simply adore donkey #Shrek pic.twitter.com/F1sawshXeQ
— Sue Atkins (@SueAtkins) March 31, 2018
Top comments (2)
I love these Twitter bots as a familiar form factor to interact with new technologies like this. It's one thing to host a webpage where people can upload an image and receive a result, but it's so much more native and fun (and shareable) to do so through a Twitter exchange like this.
Thanks for sharing. I'm going to have to give this a try.
On a related note — @nirzaq showed the process of building a Twitter bot here:
Twitter bot that generates an image with your quote!
Rizqan Arief
Thank you - I hope you have fun with it. have a look at the source code if you’re interested.