Whats your preferred method to download search results?

[soho1]

I have 65,000 search results from a search and I want to download them all.

Suggestions? er, Practical suggestions?

 

[roncerr]

...Practical suggestions?

Don't.

 

[soho1]

Don't.

Not a practical answer to the problem.

 

[rap33042]

I'm not sure what you are trying to do here, but:

If all you want is a list of search results saved to a file for future reference, you can save the page as a file on your hard drive.

If you actually want to download all the pages the results point to, then you'll need a web crawler, a whole lot of HDD space, and a whole lot more patience than HDD space. Be prepared to download a big chunk of the Internet.

 

[Layback Bear]

Can you tell us why you want to download that much of anything.

 

[sdspieg]

I have a similar request. And the reason is that I would like to textmine all of the results for various queries (e.g. to cluster, detected patterns, changes over time, etc.). Seems to me a webcrawler wouldn't help here, as that would require entering a set of websites and then crawling them. But maybe there are webcrawlers that do allow crawling only results from say a google search - in that case any further pointers would be useful.

BTW I did find a little program called Google Save Search Results from a company called Sobolsoft, but I keep getting a vb error message when I try to run it. Maybe somebody else will have more success with it.

-Stephan

 

[Mike Connor]

SysExporter: Grab data from list-view, tree-view, combo box, WebBrowser control, and text-box.

http://www.windows7download.com/win7-google-save-search-results-software/guehlicl.html

http://www.outwit.com/


Regards....Mike Connor

 

[sdspieg]

Fantastic! The second one is the one I mentioned (and I can't get to run), but I am downloading the other ones as we speak. Thanks much Mike!

-Stephan

 

[Mike Connor]

Fantastic! The second one is the one I mentioned (and I can't get to run), but I am downloading the other ones as we speak. Thanks much Mike!

-Stephan

My pleasure, hope they meet your needs.

Regards....Mike Connor

 

[sdspieg]

Outwit might. But I guess they're sleeping now as I have not received my product key yet. If anybody here has some hands-on experience in using Outwit for scraping google search results, I'd be delighted if they could share this. Essentially all I would like to do is to download the relevant parts of some 100'000 search results from google search. I am not interested in images, video, audio, etc - just the text parts in the main frame of a page.

I'd like to be able to save all of these somewhere on my hard drive in such a way that I can then textmine them in another program (which takes pretty much any format - html, doc, pdf, xml, etc.) I did find a tutorial on youtube for Outwit with google search, but that basically just shows how to extract the URLs into Excel. But so again - if anybody has any idea on how to actually save the files (and - if possible - how to scrape them), I'd be very grateful.

-Stephan

 

[Mike Connor]

I used Outwit for two projects, but I'm not sure that what I did would be of much use to you. I was looking for specific terms. with related results and related images. I just used the built in capabilities to download what I wanted automatically. I sifted the final results manually.

These may help you;

YouTube - Web Scraping a Search Results Page with OutWit Hub Data Extractor

Web Scraping a Search Engine Results Page with the OutWit Hub Data Extractor

Scrape Search Engine Result Pages (SERPS) with OutWit Hub for SEO Audit (Video) | OutWit Technologies Blog

http://blog.outwit.com/?p=121

Regards....Mike Connor

 

[Mike Connor]

Oh and by the way, the Sobol search works perfectly for me;



So I assume that you must have something set up incorrectly, wrong VB runtime or whatever.

Here's how to get the full version;

https://usd.swreg.org/cgi-bin/s.cgi?s=125581&p=125581-82274&q=1&v=0&d=0

Regards....Mike Connor

 

[soho1]

Thanks, I will try those ideas and get back to you.

Basically, my issue is that I need to download results for research and sort through them in ways that Google and Bing don't give me options to handle.

Let's use this as one example:

I'll buy you a cup of coffee (Starbucks gift card for the best qualifying answer) if someone can tell me the complete history of the "Chicken and the Pig" story, how far back it dates and how it was told (variants) before it was used in the Agile community.

As far I know, this is an Aesop's Fable, but the oldest news story I could find that referenced it was Bishop Muzorewa using the parable to describe the political situation in Rhodesia in 1977.

Along the way, you may learn some things about search that I am struggling with. Please post.