Overview

Antique Auctions was one of the first public auction websites for their niche of collectibles. They had been running with the same vendor for their website for nearly two decades, but unfortunately the vendor was going out of business.

The vendor had given notice that all data would be deleted at the end of the year, but the group of collectors managing the auction site didn't have any idea how they could download all the historical transaction data before it was deleted.

One group member had suggested finding an offshore programmer on Fiver, but many members weren't comfortable with that.

Instead, they were introduced to Cedar Labs and were impressed with how quickly they could identify the problem and quickly put together a working demo of how the crawling could be done.

A few weeks later, a download link to the raw data and an aggregated Excel document were sent to the group, and they were able to breathe a sigh of relief.

Over the course of 3 weeks, Cedar Labs delivered:

• Wrote a crawler using Selenium which crawled the entire website and saved all auction data to S3 including images, web page screenshot, raw HTML, and parsed extracted metadata
• Wrote an aggregator to iterate over all crawled data in S3 and collect it into a single Excel file
• Provided a final download link to the group for the raw data and images and the aggregated Excel file

For privacy reasons, the CEO requested anonymizing both their name and the company name in this portfolio profile.

What we did