[web-scraping] Forum Topics

Hi , I am currently doing a university project 'creation of a web mining tool to gather information about individuals and classify them semantically'. I am currently lost and don't know how to start. Does any one have tutorials links that explain how to use java to connect to website …

There is a school locator script at [url]https://www.ocps.net/parents/pages/FindaSchool.aspx[/url] When we input any address it returns schools located in that locality. For example: Use these details and submit the form Street Number : 3902 Street Name : Bobolink Street Type : Lane City : Orlando It returns a row having three …

I know the password. I want to crawl all the post of my gf's xanga down to my comp, for better reading. But my gf have set "require friend login" to see her post. So I need to login my id of xanga before reading her post. What I want …

I am crawling a website I found out that the source code of the website is different to the content of the website. obviously the content were generated through some script, does anyone know how to simulate the action and get the content of it? For example, there is a …

hi, iwant data from [url]http://www.stubhub.com/new-york-yankees-tickets/yankees-vs-astros-6-11-2010-898499/[/url] in that right side div of all pages data, how can i get, plz help me.. thanks in advance

Hello, Don't know where to post this though this topic based on web development platform, i think it may goes here. Is anyone aware of screen scrapper Enterprise. I want to use that. Anybody has any good info/practical info. Or anybody has any "video tutorial" so that i can use …

Hello and thanks for reading. I am having a bit of trouble figuring out how to collect a piece of information from a web page. Here is the web page I want to retrieve the information from: [url=http://itemdb-rs.runescape.com/results.ws?query=green+dragonhide]RuneScape - The Number 1 Free Multiplayer Game[/url] I would like to retrieve …

I have written a script for scraping a web site, and it works fine. What does not work fine, is when I try to use the write function, to write the results to a txt-file. I am trying to run this: import BeautifulSoup, urllib2, re, time import codecs path='C:/Users/Me/Documents/Python' outfile=open(r'C:/Users/Steinar/Documents/Python/Vegvesen/vegresultat.txt', …

I want take all the usernames from this webpage and output them on my site. How would I go about that? [url]http://runehead.com/clans/ml.php?clan=gotskills[/url]

Hello. I have no idea how i can copy the text from a web page to a string variable ? I dont want the HTML code of the page to be stored in the string.. i want the output of the web page to be stored into the string.. If …

Hello Does anyone know of some good Web Site Scraping software that they can reccomend Thanks

I am an undergraduate Student, from Computer Science and engineering department I can construct a crawler in Perl, for one particular web-site to fetch the useful information, in my case the - Job Ads at that company's webpage. Now, I want to construct some crawler that is generalized for say …

Hi I'm Phil Smith, CEO of Cloud Testing. We specialise in automating website testing - from the cloud. Users can upload their test scripts captured in Selenium IDE onto our service and use it to test across a number of browsers and versions. Screenshots and HTML are captured along the …

Hi, I've never used PHP before so I don't even know if I'm posting in the right place, but I'm wondering if it's possible to scrape information from a dictionary-like Web page. I want to take a list of words and retrieve piece of information about each word, but the …

I'm not sure if this is even possible (please help me if it is). But I need to make a python code that will grab a certain portion from a website, and save it temporary. Let me explain. This program audits nations (this is a game) and displays a code …

I have 2 websites: website A which is blank and is categorized; website B which doesn't have categories but has information; I want php to go to website B, get all the new information that has been added since last visit, and then update websites A database; now i know …

Hi, Is there any way to retrieve the contents of a DIV from another website, E.G. there is a div on Youtube with the ID description, how would i get the contents of that DIV? Thanx Sam Rudge

Hello everyone, This is Paryushan. I am currently working on perl and selenium and hoping to get answers to my Qs and answer some of the Qs :) Looking forward to help u and me. Thanks, Paryushan

Hi, In my project, i should display all the mail ids available in a website. one text box where i should enter the address(eg:[url]www.sample.com)and[/url] submitting it will return all the mail ids available in that website. Kindly help me out.Thanks...

I'm trying to write my first web scraper with Python using simple regular expressions to match the info I want to extract (I realize BeautifulSoup is available, but I'm not ready to use that yet, so I want to figure out how to use reg ex first) . I want …

i need to write a program which gets the entire data when i click a link in aweb page into exel sheet or into an another web page eg: i have a link ,when i click that link i get a list of names having address ,i need to get …

Hello all, I have a unique situation. I am currently trying to save a complete web page that requires you to login with a user name and password. To be more specific, here is the web page I want. [url]www.eventid.net[/url] This site requires you to have a subscription in order …

i want to take this: Label: Supreme Records Catalog#: S 512 Format: Vinyl, 7" Country: Belgium Released: 1972 Genre: Rock Style: Pop Rock Credits: Producer - Gus Dudgeon Notes: Rating: No votes yet. Rate It Submitted by: Evilist from the site : [url]http://www.discogs.com/release/479714[/url] pls show me how to code it

HI ALL, I'm interested in learning more about screen scrubing/scraping. First let me say I'm not an IT tech.. far from it. I work in the collections industry on a major federal contract. Due to my limited knowledge I have no idea where to begin (I'm actaully amazed I "found" …

I would like to write an application that would search a site like "minitorrents.com" and automatically download torrent files of tv shows I like. I am thinking urllib.py might be able to do something like this. I need to be able to use the search feature of "minitorrents.com", then recursivley …