75 Topics
I heard about the DaniWeb community from a friend and I'm very interested in it. I am a self-taught female web scraper and I'm excited to join the community. I hope to engage with everyone and share our experiences in the future!
Hi all, I was wondering, can you still get a screenshot from a url using PHP and Google Pagespeed Insights, I've tried a few codes but I just get an empty file. Any help would be cool, thanks.
How do I establish an open connection to an open web browser in C#? In a Microsoft C# program using a Visual Studio Code I am using the following namespaces: using OpenQA.Selenium; using OpenQA.Selenium.Chrome; using OpenQA.Selenium.Support.UI; After I get a string variable, "url", assigned with a URL of a website …
## Introduction ## In the [last Selenium tutorial](https://www.daniweb.com/programming/software-development/tutorials/538447/basic-interactions-with-web-elements-in-selenium), we learned how to perform basic actions such as clicking and sending keystrokes. In this tutorial, we will learn how to do another action, which is the scrolling action. ## Goals ## At the end of the tutorial, you would have learned: …
## Introduction ## Welcome to part 2 of the Selenium scrolling tutorial. Please follow the directions in [part 1](https://www.daniweb.com/programming/software-development/tutorials/538448/perform-scrolling-actions-in-selenium-part-1) to set up your project. In part 1, we learned how to let Selenium decide how it wants to scroll to elements. In this part 2, we will learn about a …
## Introduction ## When working with Selenium, you might have run into a situation where you have to wait for long-running tasks to complete before you can interact with elements. Fortunately, Selenium has support for such situations via its many *waits* API. In this tutorial, we will learn how to …
## Introduction ## When creating a Selenium test, it is common for developers to want their scripts to take the same actions as a real human user would take. These action often includes clicking on elements or inputting text data into forms. In this tutorial, we will learn how to …
## Introduction ## While creating Selenium tests, you might have found that inspecting elements manually using the web browser’s inspector could be tedious. Even after you have found the element, you still have to go through the pain of looking up the correct **xpath** syntax for this specific element. To …
## Introduction ## Selenium is one of the most popular web browser automation tools out there. There is no restriction on what you can use Selenium for. You can use it to build a GPU scalping bot (*please don’t*) or build e2e tests for your website. In this tutorial, we …
I've just started a class to learn automation testing via Selenium. I'm using Firefox 61.0.2 (64-bit) I'm noticing a possible bug; not something major, but something that prevents efficiency: When attempting to load a saved project into the IDE via Open Project or through drag-and-drop, it does not load. Usually …
Friends, Mates, Buddies & Pals, I don't want to google or youtube because there's a lot of bad tutorials rampaging the net. And so, how-about you recommending me some text & video tutorials on: CURL Hashing I really need to learn password hashing. Encrypting & decrypting. And web scraping. :thumbsup: …
import urlparse import urllib import urllib2 from bs4 import BeautifulSoup from collections import Counter import re import Tkinter from Tkinter import * import ttk from tkFileDialog import askopenfilename import logging import tkMessageBox import tkFileDialog import csv import json import os import time import collections import shutil import thread from PIL …
It's a simple script, giving me an error. I'm running it from a command line: scott$ python VerifyAboutPage.py chrome any And the error is: AttributeError: 'module' object has no attribute 'chrome' The relevant portions of the script should be: from selenium import webdriver import unittest, time, re, reporting, sys class …
I'm not sure how crazy this sounds, but bear with me. I have a long list of links that are people's names. Inside each link is a set of information, where an email address is located after other information. I need to make (or just find in general) something that …
Hi, i m new and i will sure seek for some help also i will try to help others as well with my for now limited knowledge of Python. So here is my problem: I m learnign and i want to learn how to scrape things. i did scrape some …
Hello. I have a homework. I have asked to create a web crawler that be able to enter into a music website and then for the first step, collect the name of singers that their names starts with the letter "A". Now i need a little help for this step. …
Hello, I want to create an application that downloads a table (containing a schedule) from a specific website and stores it in a text file. Then it reads the text file and performs some calculations of the schedule's time. I tried by downloading the source code of the website but …
I have been working on a project with selenium 2 webdriver for a few weeks. I got to a milestone and decided to compile my project in release mode and test the executable. Thing is, it crashes if I try to run it from the release folder (or anywhere else …
I'm having some problems cleaning up some of the processes my windows form application creates. I'm using selenium which starts a process instance of a browser and a browser driver among other things. Selenium library provides a Quit() method which if called inside my running application works just fine at …
hi i am trying to automate one form in selenium using java code. the same codee working fine in firefox but getting error in IE. Started InternetExplorerDriver server (32-bit) 2.42.0.0 Listening on port 10840 Exception in thread "main" org.openqa.selenium.NoSuchElementException: Unable to find element with name == ImpersonateUserId (WARNING: The server …
I've been working on a project using selenium 2 webdriver. Took me a while but finally got it working inside iframes of a web page. Thing is, I searched a lot and seen many examples but none that looked like what I ended up with, looks a bit weird and …
Following a tutorial [here](http://www.joecolantonio.com/2012/07/31/getting-started-using-selenium-2-0-webdriver-for-ie-in-visual-studio-c/). Code is as follows... using System; using System.Collections.Generic; using System.Linq; using System.Text; using OpenQA.Selenium; using OpenQA.Selenium.IE; using OpenQA.Selenium.Support.UI; namespace mySelenium { class Program { static void Main(string[] args) { IWebDriver driver = new InternetExplorerDriver(@"C:\Downloads\selenium"); driver.Navigate().GoToUrl("https://www.google.co.uk/"); } } } When I run the code all seems to …
I want to make site similar to [filehippo](http://www.filehippo.com),and I want to get all the content of filehippo.how can it be done?
I want to extract this data from a website `You need to add the numbers present in this string. Your string is: 3754537dd2f082ad578e0ff1806d86a6` This is what I was doing url = opener.open('http://www.website.com') data = url.read() extract = re.search('your string is: <strong>(.*)', data) ans = extract.group(1) ans = string.split(ans, '</strong>')[0] print …
hello, you can help me scrap the contents of this table? I'm interested in extracting the content of href and the text Spa. <tr bgcolor=""> <td height="32" align="center"><a class="btn btn-mini enlace_link" style="text-decoration:none;" rel="nofollow" target="_blank" title="Ver..." href="http://freakshare.com/files/a5qgdilx/Juegos2.DVDSCRHQProper.Castellano.mp4.html"><i class="icon-download"></i><b> Opcion 01</b></a></td> <td align="left"><img src="http://www.google.com/s2/favicons?domain=freakshare.com" width="16" />freakshare</td> <td align="center"><img src="http://www.yaske.net/theme/01/data/images/flags/es_es.png" width="19">Spa.</td> <td …
Dear Team, I am writing a Swing application for my project which gets the user information and saves it in a database and then displays in a JTable. I thought like, instead of manually updating the data from our client website, why not read the data from the site and …
Hey guys i need your help, i have a project "Web information extraction/gathering/scraping system" - An information extraction system extracts and gethers information from multiple websites, then; it organizes, formats and presents the information to the users. it helps users to find information on particular things (such as flight tickets, …
**SUMMARY** Hello all, I am currently working on a project that requires me to mine data from websites so i can work with it later. I have been using cURL to accomplish pretty much all the needs i have had thus far. I ran into a complication that cURL is …
Dear Everyone! I am building a website which shows in many gas stations the current price. Well, I am stuck a bit, because there are many gas stations around my country, and it wouldn't be a bright activity to keep them freshly every single day. Is there any trick or …
Hello everyone, i read all the discussion about "web scraping" here in DaniWeb forum but i didn't found a solution to my problem. I have to extract "title" and "content" of news from a website. I wrote (after reading a lot of "tutorial") these lines: $dom = new DOMDocument(); $dom->load('http://www.php.net'); …
I initialize the selenium like [CODE]public void setUp() throws Exception { setUp("http://localhost/", "*firefox"); } [/CODE] and I write my test in the method like [CODE]public void testUntitled() throws Exception { boo(); foo(); }[/CODE] But it gives error like [CODE]java.lang.RuntimeException: Could not start Selenium session: Failed to start new browser session: …
HI. I've tried other codes, but they don't seem to work. What I'm trying to do, is grab a result from google's shopping page, and display it on my page. I only want the title, which is in the tag:[CODE]<h3 class="result-title">TITLE</h3>[/CODE] How do I do that in PHP? Thanks!
Hi, I want to do program that reads different news sites and lists headlines. How this should be done? How to read the data from the web sites?
Hello allheres what im trying to do scrape a page for data between two words ,reformat it as a link and then follow that link to get the final info i want ..all works fine till the second part comes into play--following the link here is my script which produces …
Hi All, I want to build an application that will navigate a website that contains user login form and automatically logged to it and retrieve data from it's table. Is this possible? What would be my best approach here. Thanks in advance. -renzlo
hi hi want to launch a price comparison website, i was wondering if it's possible and if there is some script that i specify a few link and it automatically retrieves information and stores it and them compares it. cheers
Hello guys, I want to grab data from a website bet-mate.com. I tried some ready made software but as I understood you can grab data if there is an HTML table. In this website that I am trying to grab them there is no table. Can anyone advice me how …
Hi, I want to make a specific program and I was looking around the internet for some help. I came here but wasn't sure what section to post in...so I chose this one for now! What I want to do is make a program that uses the internet and retrieves …
Hi, i've used the Beautifulsoup module to parse the site and grab the img tag from it, but the problem is , Beautifulsoup while parsing not returning the whole content of the given url. The truncated content contain the image location I want to download: [CODE] from urllib2 import urlopen …
Hello, I'm doing a small project that aims to develop a search engine. What I need is a web crawler that can scrape backlinks for a website. I need to scratch: Title, PageText, Page Size, Back Links Does anyone know of a good web crawler that can give me that? …
Hi, I have to use Java to build a scrapper, that takes a product name and searches for that product on an e-commerce website. I understand that I will have to use an external library that would parse the html page for me, given the link. But how do I …
Ok! Hello Daniwebbers, lets see if you can help me this time: I'm planning to do a bot, to login into a site, collect some data and then send a report to me. Problem is, I've done some web-scrapping data before, but I have no idea on how to do …
Hey guys, I'm a little confused on a web scraping topic. I understand that you can use urllib2 to webscrape, but my project requires a little more advanced scraping. I want to use python to inpute data into a search box, then return the results of that information. As an …
I need a bit a help to randomly select a radio button on an page to answer questions. I am doing some Selenium / Java / webdriver development and sort of new to this. Below is a some code to find the number of radio buttons. any help that someone …
Hi people! Im having an issue, and it is really bothering me. I want to get some content (only 2 lines) from another site and paste it into my own site. The problem is that the content on the other site keeps changing, and I need it to be auto …
Hi Ok i am makeing a program and want it to check if there is an update availible by checking if a txt file with the version has a difrent number. I have been looking around and found absolutly no help for this problem. The text file would be at …
How can i get the content of a pop-up(that has no adress) from a specific website?
Hello ALL, I need to capture the HTTP requests that are sent from my web application, using Selenium and python please help me in achieving this.
Does anybody have an idea of how this data is collected: [url]http://www.boxoffice.com/statistics/analysis/most_discussed[/url] I'd like to do something similar, but I'm not sure where to start...web crawling maybe? Drue
hi i want to know how to read input from a file of xls or txt format and write output to a file using selenium tool pls help me ....
The End.