Code Snippet: Ruby Image Scraper

Ryan Baxter — Mon, 13 Aug 2007 20:46:00 -0700

I stumbled upon a screen scraping library for Ruby last week called scrAPI. It’s extremely flexible and can be seen in action on the co.mments blog post scraper. The scrAPI library can be installed by issuing the following command from your console:

gem install scrapi

Testing scrAPI was fairly easy once I figured out how to define a scraper. With that aside, I wrote a small script that saves images from a URL provided by the user. The scrAPI library could be used for good or evil, but only you can decide.

#!/usr/bin/ruby

require 'fileutils'
require 'open-uri'
require 'pathname'
require 'rubygems'
require 'scrapi'

# Get the URL input.
puts 'Enter a URL:'
url = gets.chomp

# Get the HTML source.
html = nil
open(url) {|source| html = source.read()}

# Define the scraper.
scraper = Scraper.define do
  array :images
  process "img", :images => "@src"
  result :images
end

# Scrape the HTML for images.
images = scraper.scrape(html)

# Create a directory to save the images in.
directory = url.gsub(/http:\/\//, '')
FileUtils.mkdir directory

images.each do |image_path|
  # Determine if image_path is absolute or relative. 
  path = Pathname.new(image_path)  
  if not path.relative? then image_path = url + image_path end

  # Write the image to disk.
  open(image_path) do |source|
    file_name = image_path.split('/').last
    open(directory + '/' + file_name, 'wb') {|file| file.write(source.read())}
  end
end

puts 'Finished...'

"Code Snippet: Ruby Image Scraper" by Ryan Baxter

Tue, 25 Mar 2008 20:08:54 -0700

That's great! I've gotta get back to writing more Ruby soon.

"Code Snippet: Ruby Image Scraper" by Bill

Tue, 25 Mar 2008 19:49:46 -0700

Wow. This is friggin awesome. Worked almost entirely on my first try. Many thanks!!!

crunchlife: Code Snippet: Ruby Image Scraper

Code Snippet: Ruby Image Scraper

"Code Snippet: Ruby Image Scraper" by Ryan Baxter

"Code Snippet: Ruby Image Scraper" by Bill