Testing out web scraping.
Installing
- (also works with Anaconda) @@pip install scrapy
- if error
Library not loaded: libssl.1.0.0.dylib appears, make sure the proper symlinks were created for libssl, as suggested here. In my case, the symlinks for ssl.1.0 did not exist, so I just had to run
$ sudo ln -s /Users/YOURUSERNAME/anaconda/lib/libssl.1.0.0.dylib /usr/lib
$ sudo ln -s /Users/YOURUSERNAME/anaconda/lib/libcrypto.1.0.0.dylib /usr/lib
Reference
- Scrapy documentation
- Scrapy tutorial
- For tutorial to work, items.py has to be set to the example in the tutorial.
- Also, must add
from tutorial.items import DmozItem at the top of the dmoz_spider.py
Related
