How do I download the entire BFAds.net database?

All Site Suggestions or General Black Friday Tips

Moderators: Tony, cematra173, killasnake, Huy

User avatar
BFVet
Black Friday Pro
Posts: 264
Joined: Fri Nov 04, 2011 7:41 pm

How do I download the entire BFAds.net database?

Postby BFVet » Fri Nov 18, 2011 9:56 am

I've got a spreadsheet from last year where I was able to download the entire 2010 database (store, item, price, sale time, etc.) and then sort it in Excel. It was helpful in comparing similar sales, finding items I had overlooked, etc.

Is that a feature that went away, or am I just overlooking it. Please link if you know what I'm referring to.

User avatar
Shopper X
Suspected Line Jumper
Posts: 2647
Joined: Wed Nov 16, 2005 6:05 pm

Re: How do I download the entire BFAds.net database?

Postby Shopper X » Fri Nov 18, 2011 3:21 pm

I seem to recall that it was a member of the message board that made the spreadsheet on their own, though I'm not sure.

Is this is something that people would find useful? If so, I could try writing a script that scrapes sales listings from bfads.net and outputs the information as a spreadsheet.

(bfads.net admins, please let me know if you want me to not do this. I would put delays in the script to avoid hammering the web site too much, but it's still a bit of web server traffic. Though, it probably isn't that much compared to normal web site usage, with people looking at the store listings.)

User avatar
bfaddict
Black Friday Pro
Posts: 4808
Joined: Thu Nov 16, 2006 4:19 pm
Contact:

Re: How do I download the entire BFAds.net database?

Postby bfaddict » Fri Nov 18, 2011 3:25 pm

yea someone made it last year and distributed it. i don't remember who it was though..... they werent connected to bfads at all. but it was SUPER helpful!

User avatar
Shopper X
Suspected Line Jumper
Posts: 2647
Joined: Wed Nov 16, 2005 6:05 pm

Re: How do I download the entire BFAds.net database?

Postby Shopper X » Fri Nov 18, 2011 7:44 pm

The scripts are finished. I just need to run them when I get home and fix any remaining bugs. The, I'll retrieve all the web pages and generate the spreadsheet. It should be ready tomorrow or Sunday.

The spreadsheet will be a CSV format file, following what I believe to be Excel-compatible CSV rules. Below is a sample. Currently, the fields are:
"Store, Category, Item, Price"

Code: Select all

Best-Buy,Camcorders,Kodak Playfull 1080p HD Camcorder,$69.99
Best-Buy,Camcorders,"Sony Camcorder Bundle (w/ Sony Handycam HDRCX160, Battery, 8GB Memory Card and Camcorder Bag)",$349.96
Best-Buy,Camcorders,Sony DCR-SX85/B 16GB Flash Memory Camcorder - Black,$149.99
Best-Buy,Camcorders,"Sony HDR-XR160 Bundle (w/ Camcorder, 60"" Tripod, Battery and Camcorder Bag)",$449.96
Best-Buy,eReaders & Tablets,All M-Edge eReader Covers,50% OFF
Best-Buy,Laptops,"Gateway 15.6"" Laptop w/ Intel i3 CPU, 4GB RAM, 320GB HDD (NV55C56U)",TBD
Costco,Apparel,Carter's Kids' 2-pc Microfleece Pajama,$2 Off

I didn't want to include sale/opening times because this information isn't readily available from the bfads listings. Plus some stores have items going on sale at multiple times, so listing the store opening time could be misleading and inaccurate. Is there any other information that people would like to see in the spreadsheet? The only other information that I see is available on bfads.net listings are regular price and "buy now" availability, but these are only available for some items.

Could someone please recommend a file-hosting site that I could use to share the file? I want something that doesn't require registration and that gives me the ability to remove files. (bfads.net, if you're interested in hosting the files please let me know.)

User avatar
Russell Burrows
Black Friday Pro
Posts: 722
Joined: Fri Oct 12, 2007 11:50 pm
Location: CD. Victoria, TAM, MX.
Contact:

Re: How do I download the entire BFAds.net database?

Postby Russell Burrows » Fri Nov 18, 2011 9:19 pm

Shopper X wrote:The scripts are finished. I just need to run them when I get home and fix any remaining bugs. The, I'll retrieve all the web pages and generate the spreadsheet. It should be ready tomorrow or Sunday.

The spreadsheet will be a CSV format file, following what I believe to be Excel-compatible CSV rules. Below is a sample. Currently, the fields are:
"Store, Category, Item, Price"

Code: Select all

Best-Buy,Camcorders,Kodak Playfull 1080p HD Camcorder,$69.99
Best-Buy,Camcorders,"Sony Camcorder Bundle (w/ Sony Handycam HDRCX160, Battery, 8GB Memory Card and Camcorder Bag)",$349.96
Best-Buy,Camcorders,Sony DCR-SX85/B 16GB Flash Memory Camcorder - Black,$149.99
Best-Buy,Camcorders,"Sony HDR-XR160 Bundle (w/ Camcorder, 60"" Tripod, Battery and Camcorder Bag)",$449.96
Best-Buy,eReaders & Tablets,All M-Edge eReader Covers,50% OFF
Best-Buy,Laptops,"Gateway 15.6"" Laptop w/ Intel i3 CPU, 4GB RAM, 320GB HDD (NV55C56U)",TBD
Costco,Apparel,Carter's Kids' 2-pc Microfleece Pajama,$2 Off

I didn't want to include sale/opening times because this information isn't readily available from the bfads listings. Plus some stores have items going on sale at multiple times, so listing the store opening time could be misleading and inaccurate. Is there any other information that people would like to see in the spreadsheet? The only other information that I see is available on bfads.net listings are regular price and "buy now" availability, but these are only available for some items.

Could someone please recommend a file-hosting site that I could use to share the file? I want something that doesn't require registration and that gives me the ability to remove files. (bfads.net, if you're interested in hosting the files please let me know.)



Create a torrent file with the data and upload it to any torrent site then just post the link to your uploaded torrent file :wink:

User avatar
bfaddict
Black Friday Pro
Posts: 4808
Joined: Thu Nov 16, 2006 4:19 pm
Contact:

Re: How do I download the entire BFAds.net database?

Postby bfaddict » Fri Nov 18, 2011 10:24 pm

mediafire.com maybe??
that's a good site for sharing files easily but not sure if it's what you're looking for

User avatar
BFVet
Black Friday Pro
Posts: 264
Joined: Fri Nov 04, 2011 7:41 pm

Re: How do I download the entire BFAds.net database?

Postby BFVet » Fri Nov 18, 2011 11:45 pm

thanks for doing this. looking forward to seeing the final product.

User avatar
Shopper X
Suspected Line Jumper
Posts: 2647
Joined: Wed Nov 16, 2005 6:05 pm

Re: How do I download the entire BFAds.net database?

Postby Shopper X » Sat Nov 19, 2011 2:43 am

The first draft is deals.20111118.csv with 488 files, 59 stores, 13665 items.

I'm not sure yet if the spreadsheet is absolutely correct, though the item counts for individual stores match the ones on the bfads.net home page. I'll make corrections as needed or as new ad listings are added to the bfads.net site.

I think that as updates are produced, I'll just create new files using the date in the file name. I'll also delete the old files when new ones are available. For now, I'll leave the file uncompressed, though it probably would make sense to have it as a zip or gz or bz2 file.
Last edited by Shopper X on Sat Nov 19, 2011 6:03 pm, edited 1 time in total.

User avatar
BFVet
Black Friday Pro
Posts: 264
Joined: Fri Nov 04, 2011 7:41 pm

Re: How do I download the entire BFAds.net database?

Postby BFVet » Sat Nov 19, 2011 12:00 pm

Looks good shopperX. The only challenge I faced getting it into Excel is the price column. Since the data is comma delineated, and any price over $999.99 also has a comma in it, the price column gets split for any price over $1,000. It was easy enough to fix using a formula to combine the two columns. If anyone else needs the formula, just put this in CELL F1 and copy it down column F. =IF(E1="",+D1,(+D1*1000)+E1)

josiology
Black Friday Pro
Posts: 35
Joined: Fri Nov 11, 2011 12:27 pm

Re: How do I download the entire BFAds.net database?

Postby josiology » Sat Nov 19, 2011 2:03 pm

thanks Shopper X!!!!

User avatar
Shopper X
Suspected Line Jumper
Posts: 2647
Joined: Wed Nov 16, 2005 6:05 pm

Re: How do I download the entire BFAds.net database?

Postby Shopper X » Sat Nov 19, 2011 3:34 pm

BFVet wrote:the price column gets split for any price over $1,000

Ah, that's my mistake. I didn't think that there would be any commas in the price field, so I didn't apply the quotation mark delimiter stuff to it. I guess it shows what sort of cheapo I am that I didn't consider that there might be things that cost more than $1,000. I'll fix it in the next revision of the spreadsheet.

User avatar
Shopper X
Suspected Line Jumper
Posts: 2647
Joined: Wed Nov 16, 2005 6:05 pm

Re: How do I download the entire BFAds.net database?

Postby Shopper X » Sat Nov 19, 2011 6:10 pm

The second revision is deals.20111119.csv with 498 files, 63 stores, 13972 items.

It contains 4 new stores added since yesterday (Amazon, Northern Tool, Olympia Sports, True Value). It also contains the fix for the "commas in prices" problem.

I've found that the bfads.net item listings change daily even if no stores are added. The listings are constantly updated with corrected information, and in a few instances there are new items added (probably ones that were missing before). The upshot for me is that it is necessary to reload all the web pages each time the spreadsheet is generated to make sure the information is current.
Last edited by Shopper X on Sun Nov 20, 2011 11:16 pm, edited 1 time in total.

User avatar
BFVet
Black Friday Pro
Posts: 264
Joined: Fri Nov 04, 2011 7:41 pm

Re: How do I download the entire BFAds.net database?

Postby BFVet » Sat Nov 19, 2011 6:51 pm

13972 items

Is anyone else shocked by that number? I guess those 64 page ads from Sears, JC Penny and Kohls add up quickly.

User avatar
Shopper X
Suspected Line Jumper
Posts: 2647
Joined: Wed Nov 16, 2005 6:05 pm

Re: How do I download the entire BFAds.net database?

Postby Shopper X » Sun Nov 20, 2011 11:14 pm

(I'll keep updating this post as new revisions of the spreadsheet are available.)

Revision 3 is deals.20111120.csv with 504 pages, 63 stores, 14150 items.
Revision 4 is deals.20111121.csv with 505 pages, 64 stores, 14178 items.
Revision 5 is deals.20111122.csv with 505 pages, 64 stores, 14178 items.
Revision 6 is deals.20111123.csv with 527 pages, 70 stores, 14701 items.
Revision 7 is deals.20111124.csv with 527 pages, 70 stores, 14701 items.
Last edited by Shopper X on Thu Nov 24, 2011 10:37 pm, edited 1 time in total.

User avatar
Shopper X
Suspected Line Jumper
Posts: 2647
Joined: Wed Nov 16, 2005 6:05 pm

Re: How do I download the entire BFAds.net database?

Postby Shopper X » Wed Nov 23, 2011 9:56 pm

Just a bump to remind people that a spreadsheet containing all the Black Friday deals is available. I've been updating it daily to include newly-added deals, like the ones in the Home Depot ad that were just added tonight. The link to the downloadable spreadsheet is in the post above.


Return to “Suggestions and Tips”

Who is online

Users browsing this forum: No registered users and 1 guest