tag:blogger.com,1999:blog-1699790186532417278.post347385791220156915..comments2023-06-17T07:40:37.867+02:00Comments on theBioBucket*: R-Function GScholarScraper to Webscrape Google Scholar Search ResultUnknownnoreply@blogger.comBlogger16125tag:blogger.com,1999:blog-1699790186532417278.post-58989833457632168132011-11-14T00:24:03.900+01:002011-11-14T00:24:03.900+01:00Kay
Glad it worked for you too! It also looked th...Kay<br /><br />Glad it worked for you too! It also looked the same to me when I compare your 'Regualar Expression' approach again my XPath approach (with stem = FALSE).<br /><br />At your suggestion, I have adapted a couple of the original comments to reflect some of the changes I made, such as replacing a For loop with a vectorised alternative (I should have done that before, thanks for point it out!)<br /><br />BTW, I notice that you are using google docs as a way to "group-edit" R scripts you've produced. Have you thought about github instead? I'm still very new to it but as I understand it, group editing is one of its features.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1699790186532417278.post-80941933714253992342011-11-13T22:32:11.609+01:002011-11-13T22:32:11.609+01:00Tony,
nicely done! ..ran some search strings and i...Tony,<br />nicely done! ..ran some search strings and it looks good to me - I can't really comment the code - all the xpath stuff is beyond my horizon.<br /><br />For the searches I tried I got equal results (as far as I can tell..) as with my function..<br /><br />One thing: maybe the original commentary should be adapted in some places?Kayhttps://www.blogger.com/profile/09320307979146470105noreply@blogger.comtag:blogger.com,1999:blog-1699790186532417278.post-268515578709875522011-11-13T19:56:19.039+01:002011-11-13T19:56:19.039+01:00Kay, I made an XPath version of your function, now...Kay, I made an XPath version of your function, now called GScholarXScraper. Full code is here:<br /><br />https://github.com/tonybreyal/Blog-Reference-Functions/blob/master/R/GScholarXScraper/GScholarXScraper.R<br /><br />I'll write a blog about it in the next few days, just wanted to let you see what I'd done with it and if you hand any comments? I wanted to make sure I gave credit correctly, hopefully I did!<br /><br />This was fun, cheers for making your code public :)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1699790186532417278.post-89929845383215546182011-11-11T21:08:49.433+01:002011-11-11T21:08:49.433+01:00Sure, you're welcome - i'm itching to see ...Sure, you're welcome - i'm itching to see how you pimped my function..Kayhttps://www.blogger.com/profile/09320307979146470105noreply@blogger.comtag:blogger.com,1999:blog-1699790186532417278.post-3845973792092496962011-11-11T15:53:06.629+01:002011-11-11T15:53:06.629+01:00Kay,
If it's any comfort, it took me quite a w...Kay,<br />If it's any comfort, it took me quite a while to understand how to use the *apply family of functions, but they're quite easy once you get the hang of them!<br /><br />Have you thought about putting your code on github? I've just set up an account about an hour ago and it's quite impressive. <br /><br />Here's my hack of your function so far:<br /><br />https://github.com/tonybreyal/Blog-Reference-Functions/blob/master/R/get_google_scholar_webpages.R<br /><br />Eventually, if I get time, I'll incorporate everything your function does. I hope it's OK that I've basically copied and pasted it (I give credit in the file)? :)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1699790186532417278.post-69401406228045805172011-11-11T14:41:16.249+01:002011-11-11T14:41:16.249+01:00Tony,
many thanks for your worthily comments! your...Tony,<br />many thanks for your worthily comments! your solution ("Solving by supplying a vector of URLs") sounds perfect!<br /><br />You see I'm a lousy programer - apply is not my friend yet - but I hope it will be soon..<br /><br />Many thanks for the hint:<br />webpages <- lapply(urls, getURL)<br /><br />@Anonymous (M),<br />many thanks for the tip with the locale!!Kayhttps://www.blogger.com/profile/09320307979146470105noreply@blogger.comtag:blogger.com,1999:blog-1699790186532417278.post-13669636660612894212011-11-11T12:35:11.018+01:002011-11-11T12:35:11.018+01:00# @Kay - I think you could do this for the webpage...# @Kay - I think you could do this for the webpages bit, getting rid of the for loop entirely.<br /><br />get_GS_pages <- function(search.str) {<br /> require(RCurl)<br /> require(stringr)<br /> <br /> # Initial URL<br /> url <- paste("http://scholar.google.com/scholar?start=0&q=", search.str, <br /> "&hl=en&lr=lang_en&num=100&as_sdt=1&as_vis=1",<br /> sep = "")<br /> <br /> # ...we’re using urls like: http://scholar.google.com/scholar?start=0&q=allintitle:+amphibians+richness+OR+diversity&hl=en&lr=lang_en&num=100&as_sdt=1&as_vis=1<br /> html_str <- getURL(url)<br /> <br /> # Find html place holders (2 alternatives!) for number of results,<br /> # and pull the number.<br /> # (!) Strangely Google Scholar gives different numbers of results<br /> # dependent on start value.. i.e., a change from 900 to 980 results<br /> # when changing start = 0 to start = 800<br /> match_no.res <- str_match(html_str, "Results <b>1</b> - <b>(.*?)</b> of <b>(.*?)</b>")<br /> no.res <- match_no.res[1, max(dim(match_no.res))]<br /> <br /> # stop if no search results found<br /> if(length(no.res) == 0 | is.na(no.res)){<br /> match_no.res <- str_match(html_str, "Results <b>1</b> - <b>(.*?)</b> of about <b>(.*?)</b>")<br /> no.res <- match_no.res[1, max(dim(match_no.res))]<br /> }<br /> <br /> # Remove punctuation (Google uses decimal commas):<br /> no.res <- as.integer(gsub("[[:punct:]]", "", no.res))<br /> <br /> # If there are no results, stop and throw an error message:<br /> if(length(no.res) == 0 | is.na(no.res)){stop("\n\n...There is no result for the submitted search string!")}<br /> <br /> # Define number of pages with results to be used subsequently<br /> # pages.max = maximum number of pages (chunk with 100 results each)<br /> # to be submitted subsequently.<br /> # Above it was said that no.res varies, depending on start value.<br /> # However, we use ceiling and the change will very unlikely be greater<br /> # than 100, so we may also add one page plus, to be save:<br /> pages.max <- ceiling(as.integer(no.res)/100)+1<br /> <br /> # "start" as used in url, defines the i-th result to start the page with<br /> # start = 0 was already used above so we need 100, 200, ...<br /> start <- c(100*1:(pages.max-1))<br /> <br /> # Collect webpages as list, the first was already retrieved and is assigned to first<br /> # list-element. the rest will be assigned in th below for loop:<br /> urls <- paste("http://scholar.google.com/scholar?start=", start[(2:pages.max)-1],<br /> "&q=", search.str,<br /> "&hl=en&lr=lang_en&num=100&as_sdt=1&as_vis=1", <br /> sep = "")<br /> <br /> webpages <- lapply(urls, getURL)<br /> <br /> # return webpages<br /> return(c(html_str, webpages))<br />}<br /><br />search.str <- "allintitle:+amphibians+richness+OR+diversity"<br />webpages <- get_GS_pages(search.str)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1699790186532417278.post-90105305327284605442011-11-11T12:31:21.608+01:002011-11-11T12:31:21.608+01:00@Kay One way around that with my function would be...@Kay One way around that with my function would be to supply a vector of google scholar urls as follows:<br /><br />df = do.call("rbind",<br /> lapply(gs.urls, get_google_scholar_df))<br /><br />this would produce an aggregate dataframe of information from all the pages provided.<br /><br />I really like the wordclouds you've produced (I had no idea how to do those before). What I might do is take your function and modify it to accept one of my scraping functions to make it work not only with Google Scholar but also search results from websites like bing.com, yahoo.com and google.com, just for fun :)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1699790186532417278.post-5849595229548706992011-11-11T10:25:29.057+01:002011-11-11T10:25:29.057+01:00I also hit the substring error. Try this: Sys.setl...I also hit the substring error. Try this: Sys.setlocale(locale="C")<br />Cheers, MAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-1699790186532417278.post-70104439590076936822011-11-11T09:42:59.189+01:002011-11-11T09:42:59.189+01:00Anonymous,
I'm afraid you'll need some R f...Anonymous,<br />I'm afraid you'll need some R first: see, i.e., the links here: http://thebiobucket.blogspot.com/p/starter_19.html<br /><br />Tony,<br />XML is to prefer because it is more systematic than my picking from strings.. One drawback of your function is that it only retrieves results of the first page.. But I guess there is a way to solve that. Maybe I will remix yours and mine when there is time for it!Kayhttps://www.blogger.com/profile/09320307979146470105noreply@blogger.comtag:blogger.com,1999:blog-1699790186532417278.post-11720656118538970002011-11-10T15:33:19.247+01:002011-11-10T15:33:19.247+01:00lol, yeah I thought it was funny too :) Although I...lol, yeah I thought it was funny too :) Although I personally prefer the XPath approach using the XML package, I've learned quite a bit from your code about stringr which, to me at least, looks like a really cool package manipulating text strings. Good work!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1699790186532417278.post-63790681481805230822011-11-10T11:36:15.322+01:002011-11-10T11:36:15.322+01:00I am very new to this I need to "scrap?"...I am very new to this I need to "scrap?" names of articles, author-journal-date (is it possible to sepparate?), and number of citations into spreadsheet (i guess XML?).<br />I was not able to find any gscholarscrapper. And i still wonder were i should put The Code.<br />Please help!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1699790186532417278.post-65798277471777634392011-11-10T09:12:41.882+01:002011-11-10T09:12:41.882+01:00Tony,
I read your post - funny that we came up wit...Tony,<br />I read your post - funny that we came up with the same thing at the very same day!<br />I replaced readLines() by getURL() - thanks for reporting this!<br /><br />Sean,<br />Sorry, at the moment I can not reproduce the substring() error. I need to see what it means..<br /><br />Anonymous,<br />it was fun for me doing it - nevertheless, the purpose is evident, without fun.. In scientific research it's vital to know what is going on, that is, to know what's being published!<br />My, or similar tools could be a very use*ful* in this regard!Kayhttps://www.blogger.com/profile/09320307979146470105noreply@blogger.comtag:blogger.com,1999:blog-1699790186532417278.post-36079608706511526452011-11-09T20:29:14.349+01:002011-11-09T20:29:14.349+01:00This is a usefool / fun project
but simply runnin...This is a usefool / fun project<br /><br />but simply running your code w/ your example i'm getting this error<br /><br /><br />Error in substring(string, start, end) : <br /> invalid multibyte string at ' * Wi<6c>dlife <Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1699790186532417278.post-15033247175370059832011-11-09T20:12:22.149+01:002011-11-09T20:12:22.149+01:00When I try to run the script I get the message &qu...When I try to run the script I get the message "Error in substring(string, start, end) : invalid multibyte string at ..."Anonymoushttps://www.blogger.com/profile/12864932778225948282noreply@blogger.comtag:blogger.com,1999:blog-1699790186532417278.post-84885921196563554282011-11-09T18:09:40.875+01:002011-11-09T18:09:40.875+01:00Interesting, I'm going to have to find some ti...Interesting, I'm going to have to find some time to try your function out! As an exercise in XPath, I used the XML package to scrape information off Google Scholar and return it into a data frame. You've gone the regular expression path which I'll have to try and find some time to understand (looks good though!)<br /><br />Running the script, I hit an error:<br /><br />[sourcecode language="r"]<br />search.str <- "allintitle:+amphibians+richness+OR+diversity"<br /><br />url <- paste("http://scholar.google.com/scholar?start=0&q=", search.str, "&hl=en&lr=lang_en&num=100&as_sdt=1&as_vis=1", sep = "")<br /><br />webpage <- readLines(url, warn = F)<br />html_str <- paste(webpage, collapse="\n")<br /><br /># <br />[/sourcecode]<br /><br />However I think the following will do the same thing:<br />[sourcecode language="r"]<br />library(RCurl)<br />html <- getURL(url)<br />[/sourcecode]<br /><br />tested on R2.14.0 on Ubuntu 11.10 x64Anonymousnoreply@blogger.com