This snippet gets tweets, filters their links, sorts, resolves and counts them in about 30 lines Clojure, showing off the power of its Java interoperability and concurrent data structures.
(ns news
(:refer-clojure :exclude [resolve])
(:use clojure.contrib.json)
(:import [java.io BufferedReader InputStreamReader]
[java.net URL]))
(def cred "") ; username:password
(def json (let [con (.openConnection (URL. "http://stream.twitter.com/1/statuses/sample.json"))]
(.setRequestProperty con "Authorization" (str "Basic "
(.encode (sun.misc.BASE64Encoder.) (.getBytes cred))))
(BufferedReader. (InputStreamReader. (.getInputStream con)))))
(def urls (agent (list)))
(def resolve (memoize (fn [url]
(try
(let [con (doto (.openConnection (URL. url))
(.setInstanceFollowRedirects false)
(.connect))
loc (.getHeaderField con "Location")]
(.close (.getInputStream con))
(if loc loc url))
(catch Exception _ url)))))
(defn top [urls]
(reduce #(if (> (val %1) (val %2))
%1 %2)
(frequencies @urls)))
(future (doseq [tweet (repeatedly #(read-json json))
url (:urls (:entities tweet))]
(send-off urls #(conj % (resolve (:url url))))))
And this is just the start. I want to see if I can use Aleph and ClojureQL to kickstart my own little news service with it.
