New
#1
Whats your preferred method to download search results?
I have 65,000 search results from a search and I want to download them all.
Suggestions? er, Practical suggestions?
Whats your preferred method to download search results?
-
-
-
New #3
-
New #4
I'm not sure what you are trying to do here, but:
If all you want is a list of search results saved to a file for future reference, you can save the page as a file on your hard drive.
If you actually want to download all the pages the results point to, then you'll need a web crawler, a whole lot of HDD space, and a whole lot more patience than HDD space. Be prepared to download a big chunk of the Internet.
-
-
New #6
I have a similar request. And the reason is that I would like to textmine all of the results for various queries (e.g. to cluster, detected patterns, changes over time, etc.). Seems to me a webcrawler wouldn't help here, as that would require entering a set of websites and then crawling them. But maybe there are webcrawlers that do allow crawling only results from say a google search - in that case any further pointers would be useful.
BTW I did find a little program called Google Save Search Results from a company called Sobolsoft, but I keep getting a vb error message when I try to run it. Maybe somebody else will have more success with it.
-Stephan
-
-
New #8
Fantastic! The second one is the one I mentioned (and I can't get to run), but I am downloading the other ones as we speak. Thanks much Mike!
-Stephan
-
-
New #10
Outwit might. But I guess they're sleeping now as I have not received my product key yet. If anybody here has some hands-on experience in using Outwit for scraping google search results, I'd be delighted if they could share this. Essentially all I would like to do is to download the relevant parts of some 100'000 search results from google search. I am not interested in images, video, audio, etc - just the text parts in the main frame of a page.
I'd like to be able to save all of these somewhere on my hard drive in such a way that I can then textmine them in another program (which takes pretty much any format - html, doc, pdf, xml, etc.) I did find a tutorial on youtube for Outwit with google search, but that basically just shows how to extract the URLs into Excel. But so again - if anybody has any idea on how to actually save the files (and - if possible - how to scrape them), I'd be very grateful.
-Stephan
Quote
Related Discussions