Saturday, July 31, 2021

selenium scroll

 his is a HTML :

reference: https://michaeljsanders.com/2017/05/12/scrapin-and-scrollin.html

import time
from selenium import webdriver
from bs4 import BeautifulSoup as bs
# I used Firefox; you can use whichever browser you like.
browser = webdriver.Chrome()
# Tell Selenium to get the URL you're interested in.
browser.get("http://URLHERE.com")
# Selenium script to scroll to the bottom, wait 3 seconds for the next batch of data to load, then continue scrolling.  It will continue to do this until the page stops loading new data.
lenOfPage = browser.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")
    match=False
        while(match==False):
                lastCount = lenOfPage
                time.sleep(3)
                lenOfPage = browser.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")
                if lastCount==lenOfPage:
                    match=True
# Now that the page is fully scrolled, grab the source code.
source_data = browser.page_source
# Throw your source into BeautifulSoup and start parsing!
bs_data = bs(source_data)




Sunday, July 11, 2021

conda not found in MAC

 Have you ever see an issue in MAC when Anaconda is been installed propely and the conda is place correctly but still conda will be not found. I'm going to teach you how to solve this issue: 


Part1 In case you don't know how to add CONA to PATH

Note: Mac Anaonda is been save in /opt/anaconda3

1. Edit vim ~/.bash_profile

export PATH="/opt/anaconda3/bin:$PATH"

2. wq! or x! to save it 

3. source ~/.bash_profile

4. you can check PATH  by:   echo $PATH

conda --version

6 open a new terminal and enter conda --version will occur command not found 

Part 2 Terminal running conda 

To solve this issue we basely have to just touh a file. In MAC default it uses zshrc

Method1:  

1. touch ~/.zshrc 

2. edit it 

source ~/.bash_profile

3. save it by wq! or x!

4. open new terminal and press coda--version will fix this problem


Method2: 

1. go to system preferences>user&groups> 

2. unlock your setting 

3 click on a username and right-click and select advance option

4. login shell changed to /bin/bash will fix this