2016 Spreadsheet of all the Black Friday deals

All Site Suggestions or General Black Friday Tips

Moderators: Tony, Huy, cematra173, killasnake

User avatar
Shopper X
Suspected Line Jumper
Posts: 2637
Joined: Wed Nov 16, 2005 6:05 pm

2016 Spreadsheet of all the Black Friday deals

Postby Shopper X » Wed Oct 12, 2016 11:26 pm

This is a spreadsheet that you can view and download. It contains all the 2016 Black Friday deals listed on BFAds.net.

November 24, 2016, 5.2 MB, 136 ads, 31,380 items

November 23, 2016, 5.2 MB, 136 ads, 31,357 items
November 22, 2016, 5.1 MB, 133 ads, 30,971 items
November 21, 2016, 5.0 MB, 127 ads, 30,342 items
November 20, 2016, 4.9 MB, 122 ads, 29,991 items
November 19, 2016, 4.8 MB, 116 ads, 29,265 items
November 18, 2016, 4.6 MB, 110 ads, 28,220 items
November 17, 2016, 4.3 MB, 99 ads, 26,284 items
November 16, 2016, 4.2 MB, 96 ads, 25,575 items
November 15, 2016, 3.8 MB, 88 ads, 23,583 items
November 14, 2016, 3.4 MB, 78 ads, 21,220 items
November 13, 2016, 3.4 MB, 72 ads, 20,961 items
November 11, 2016, 3.0 MB, 68 ads, 18,586 items
November 10, 2016, 2.3 MB, 54 ads, 14,182 items
November 9, 2016, 1.8 MB, 46 ads, 11,290 items
November 8, 2016, 1.4 MB, 42 ads, 8,321 items
November 6, 2016, 1.1 MB, 35 ads, 7030 items
November 5, 2016, 1.0 MB, 30 ads, 6174 items
November 4, 2016, 0.9 MB, 23 ads, 5776 items
November 3, 2016, 0.5 MB, 14 ads, 2717 items
October 26, 2016, 65.9 KB, 3 ads, 409 items
October 13, 2016, 13.4 KB, 1 ad, 69 items

The spreadsheet is a plain text file in CSV format that can be imported into Excel or other spreadsheet programs. The information is pulled from the BFAds.net web site, so any errors in the web site listings will also show up in the spreadsheet. Please let me know if you have any questions or suggestions or if there are any problems. The spreadsheet columns are: "Store,Category,Item,Price,URL". For historic research purposes, the 2015, 2014, 2013, 2012, and 2011 spreadsheets are also available.

(I'll keep updating this top post as updated versions of the spreadsheet become available.)
Last edited by Shopper X on Thu Nov 24, 2016 1:48 pm, edited 33 times in total.

User avatar
lwmjr
Evil, demented & twisted
Posts: 12987
Joined: Wed Jul 31, 2013 2:46 pm
Location: Virginia, Norfolk area

Re: 2016 Spreadsheet of all the Black Friday deals

Postby lwmjr » Thu Oct 13, 2016 8:48 am

Bad idea, you are just inviting people to steal from you
Just here to have fun, keep the bad guys out, and give the stray dogs a place to stay

User avatar
Shopper X
Suspected Line Jumper
Posts: 2637
Joined: Wed Nov 16, 2005 6:05 pm

Re: 2016 Spreadsheet of all the Black Friday deals

Postby Shopper X » Tue Nov 01, 2016 8:27 am

The spreadsheet is currently on hold. BFAds.net recently changed how their deals web pages work, and so my scripts no longer work.

What changed is that you can no longer view all the deal listings by simply browsing web pages. Instead, only an initial 10 deals are shown, and there is a "View More Deals" button at the bottom of the web page which uses Javascript to dynamically load additional deals to the page, 10 at a time. I would need to change how my scripts work to be able to "click" that button multiple times to load all the deals before I can access the information.

I'm not sure if I'll be able to solve the problem. I have some ideas, but the task might be difficult.

User avatar
bfaddict
Black Friday Pro
Posts: 4808
Joined: Thu Nov 16, 2006 4:19 pm
Contact:

Re: 2016 Spreadsheet of all the Black Friday deals

Postby bfaddict » Thu Nov 03, 2016 10:36 am

Shopper X - was it you or someone else that would make the PDFs of the ad scans in previous years?

User avatar
lwmjr
Evil, demented & twisted
Posts: 12987
Joined: Wed Jul 31, 2013 2:46 pm
Location: Virginia, Norfolk area

Re: 2016 Spreadsheet of all the Black Friday deals

Postby lwmjr » Thu Nov 03, 2016 11:06 am

bfaddict wrote:Shopper X - was it you or someone else that would make the PDFs of the ad scans in previous years?

it was hiker
Just here to have fun, keep the bad guys out, and give the stray dogs a place to stay

User avatar
bfaddict
Black Friday Pro
Posts: 4808
Joined: Thu Nov 16, 2006 4:19 pm
Contact:

Re: 2016 Spreadsheet of all the Black Friday deals

Postby bfaddict » Thu Nov 03, 2016 2:36 pm

lwmjr wrote:
bfaddict wrote:Shopper X - was it you or someone else that would make the PDFs of the ad scans in previous years?

it was hiker


ahhh thanks!! I miss the PDFs.
I suppose I could make my own.... but I feel too lazy lol

User avatar
CrazyTrain
Black Friday Pro
Posts: 506
Joined: Sun Nov 15, 2009 3:39 pm

Re: 2016 Spreadsheet of all the Black Friday deals

Postby CrazyTrain » Thu Nov 03, 2016 7:58 pm

bfaddict wrote:
lwmjr wrote:
bfaddict wrote:Shopper X - was it you or someone else that would make the PDFs of the ad scans in previous years?

it was hiker


ahhh thanks!! I miss the PDFs.
I suppose I could make my own.... but I feel too lazy lol

Me too. Its the easier way for me to make my list.
They used to do it on the site, but i don't think they have for awhile.

User avatar
Shopper X
Suspected Line Jumper
Posts: 2637
Joined: Wed Nov 16, 2005 6:05 pm

Re: 2016 Spreadsheet of all the Black Friday deals

Postby Shopper X » Thu Nov 03, 2016 8:53 pm

I did make a few PDF files in the past. However, I had to do it somewhat manually, making it a tedious task. The scripts I previously used to download adscan images stopped working when bfads.net was updated. I need to fix my scripts in order to get the adscans again.

I found a new way to download bfads.net deals and generate the spreadsheet again. The "Early Bird" and "Rebate" info is no longer provided, so I removed them from the spreadsheet. If you see any problems with the new spreadsheets, please let me know. I'm still refining the process.

There is a bug on bfads.net such that all the deals are not displayed. For example, if there are 94 deals for a store, only 90 get displayed. If there are 52 deals, only 50 get displayed, etc. My spreadsheet includes all the deals I was able to extract.

(BFAds.net, as always if you want me to stop making this spreadsheet (since I'm basically taking/stealing all the data from your web site), please just let me know.)

User avatar
Shopper X
Suspected Line Jumper
Posts: 2637
Joined: Wed Nov 16, 2005 6:05 pm

Re: 2016 Spreadsheet of all the Black Friday deals

Postby Shopper X » Sun Nov 06, 2016 4:47 pm

I think that I've finally gotten a handle on how to generate the spreadsheet, adapting to the recent changes in how bfads.net works.

Downloading deal listings
To download the deal listings, I now use CasperJS and PhantomJS. Together they basically simulate a web browser, including running the javascript that dynamically loads the content after the web page is initially retrieved. In the simulated web browser, the script is able to "press" the "View More Deals" button multiple times to run the javascript to load more deals until all the deals show up on the simulated web page. Then, the script is able to find all the elements in the web page to extract the item names, titles, prices, and URLs. Currently, it takes about 20-21 minutes to download all the deals.

The missing deals problem
There is a bug on bfads.net where if there are more than 20 deals to be retrieved, the page retrieves additional deals only in groups of 10. So, if there are 64 deals total, the page will only show you 60. If there are 39 deals total, the page will only show you 30.

The duplicate deals problem
There is another bug on bfads.net where some of the deals in the listing are duplicates. So, if there are 64 deals total, the page might show you only 60. But, 5 of them are duplicates, so you are really only seeing 55 unique deals!

The random deals problem
There is a behavior of bfads.net where the deals you are presented are often in a different order when you reload the web page. So, some deals you previously saw are now missing. Some of the previously-missing deals are now seen. You get a different set of the duplicate deals, etc.

The other duplicate deals problem
There is another behavior of bfads.net where the admins are constantly updating and correcting the individual deal listings. So, one day an item name might be just "Winter Mittens". The next day it will be "Winter Mittens (Assorted Styles)". One day a price might be "Up To 55% OFF", and the next day it is "UP TO 55% OFF" (different capitalization). Frequently when an item name changes, the URL also changes. If you are just looking at item names or URLs, you might be tricked into thinking these are different items when they are actually duplicates.

Solution to the missing/random deals problem
When you pull the all deals from bfads.net, you get a set of items with random ones missing. However, each time you pull all the deals, you get a different random set that includes some of the previously-missing items. So, if you keep pulling deals over and over again enough times, eventually you will see most or all the deals at least once. Each time we pull the deals, we save the results in a file. By combining multiple files from multiple past runs (for example, the 10 most recent past runs), we can greatly reduce the number of missing deals. Hopefully, this might even be enough so we are no longer missing any deals.

Solution to the duplicate deals problems
Since we are combining multiple files, we need to eliminate the duplicates we get from different runs. Doing this also solves the problems of duplicates seen in the same run and items with changing titles or URLs. Luckily, each item has a unique item number which we can get by looking at the item's URL. Even if an item's name, price, or URL changes, the item number will remain the same. So, by keeping track of the item numbers previously seen, we can eliminate duplicate deals. By keeping only the most recent deal for a given item number, we can make sure we have the most up-to-date entry for that item.

Limitations
If the admins delete an item and then re-add it using a different item number, it will show up as two different items in my spreadsheet. Or, if the admins delete an item altogether, it will still show up in the spreadsheet since it will be still in the files of most recently-seen deals. The spreadsheet will include these old and deleted items until enough time has passed for the items to no longer appear in the most recent saved files.

glenmillsshopping
Black Friday Newbie
Posts: 1
Joined: Sat Nov 12, 2016 9:28 pm

Re: 2016 Spreadsheet of all the Black Friday deals

Postby glenmillsshopping » Sat Nov 12, 2016 9:31 pm

Shopper X - can you please send me userid and password for Excel - glenmillsshopping@gmail.com

User avatar
jeanerz13
Black Friday Pro
Posts: 1723
Joined: Fri Oct 15, 2010 10:42 pm
Location: Missouri

Re: 2016 Spreadsheet of all the Black Friday deals

Postby jeanerz13 » Sun Nov 13, 2016 12:57 am

glenmillsshopping wrote:Shopper X - can you please send me userid and password for Excel - *snip*

I wouldn't recommend putting your email in a public forum...

srivenkat
Black Friday Newbie
Posts: 1
Joined: Sun Nov 13, 2016 10:28 pm

Re: 2016 Spreadsheet of all the Black Friday deals

Postby srivenkat » Sun Nov 13, 2016 10:31 pm

please share the credential to access the excel file. Thanks

vasdella
Black Friday Newbie
Posts: 1
Joined: Mon Nov 14, 2016 12:37 am

Re: 2016 Spreadsheet of all the Black Friday deals

Postby vasdella » Mon Nov 14, 2016 12:39 am

Can you please send me the username and pwd.

User avatar
jeanerz13
Black Friday Pro
Posts: 1723
Joined: Fri Oct 15, 2010 10:42 pm
Location: Missouri

Re: 2016 Spreadsheet of all the Black Friday deals

Postby jeanerz13 » Mon Nov 14, 2016 12:53 am

You guys know you can send him a direct message...

ashwinmandha
Black Friday Newbie
Posts: 1
Joined: Mon Nov 14, 2016 12:41 am

Re: 2016 Spreadsheet of all the Black Friday deals

Postby ashwinmandha » Mon Nov 14, 2016 1:15 am

Hi,

Can you please share Username and password to Black Friday deals Spreedsheet.

Thanks


Return to “Suggestions and Tips”

Who is online

Users browsing this forum: No registered users and 1 guest