Client Angie
Python programs that handle 5,000 to 10,000 Twitter and LinkedIn profiles per study, a volume nobody covers by hand.
- Year
- 2020 — 2021
- Client
- Angie · communications agency
- Stack
- Python · Scraping · Twitter · LinkedIn
Context
Angie is a communications agency. Its data team advises public figures, political leaders and companies on their communication. To advise them, the team needs to know what is being said on a subject, who is saying it, and to how many people.
That information is public, but scattered across Twitter and LinkedIn, one profile at a time. A study could cover 5,000 to 10,000 profiles. By hand, that volume is out of reach: without a program to do it for the team, the study simply does not happen.
My role
My first freelance mission, won after an internship in the same team. I was alone on it. I wrote the programs that collect the information, cleaned the data and put it in a single format, then automated the steps that came back with every study.
Approach
Everything was built in Python.
Speed first
Handled one after another, 10,000 profiles take far too long. I measured where that time was going, then had the program handle several profiles at once. That is what makes a study of this size fit the agency's deadlines.
Reliability next
Twitter and LinkedIn limit automated collection and change their pages often. So the program spaces out its requests, restarts where it stopped after an interruption, and I adapted it whenever a page changed.
Quality last
Collected information arrives messy. I removed duplicates and aligned the formats of both networks, so the team worked from one clean file instead of two exports to stitch together.
Outcome
Studies the data team could not run became possible. Analysing 5,000 profiles was never a matter of time, it was out of reach.
The steps repeated on every study run without anyone touching them, and the team receives data it can use straight away.