How to automate Instagram engagements with computer vision (and get banned)

4 hours ago

Instagram's HTML and class names change frequently, making DOM-based automation unreliable.
UI elements like heart icons remain visually consistent for human users, allowing computer vision to locate them.
Taking a screenshot and using template matching to find UI elements bypasses the need for DOM selectors.
Searching the entire screen leads to many false positives due to the large number of pixels.
Using landmarks like the triple-dots menu and action bar to narrow the search space improves accuracy.
Hearts appear in a consistent vertical column; filtering detections based on this alignment removes false positives.
A detection pipeline involves taking a screenshot, finding landmarks, cropping, template matching, and filtering.
Despite attempts to mimic human behavior, automated accounts may still be banned by Instagram's bot detection.
The experiment demonstrates that computer vision and cursor automation can interact with any screen-based interface.

Hasty Briefsbeta