Geocoding part 1 - Getting the longitude and latitude of all australian postcodes from google maps
8 Comments Published August 26th, 2008 in codeAs part of a grand plan to be able to map a bunch of addresses onto a graphical map of sydney or australia, firstly you need all the australian postcodes and their longitude and latitude. Now in this article i describe how to screen scrape this list from google maps (please forgive me Mr Google!).
Firstly you'll need the list of all postcodes from Australia Post. You can get it here - it is a zip of a CSV file. It contains a whole bunch of non-physical addresses such as PO boxes - if you're inclined, open it in Excel and remove these. What you want to end up with is a file called 'pc-full.csv' which we'll use later.
For the next step you'll need Curl installed. You can get it here. If you're behind a firewall, there's some hints here on how to get Curl/Wget to play nicely with firewalls (even though the instructions are for wget, it still applies to curl). Once you think you've got it sorted, try this from the command prompt:
curl "http://maps.google.com.au/maps?f=q&hl=en&geocode=&q=2768+australia&output=js"
You should get a printout of a whole bunch of javascript with 'center:{lat:xxx,lng:yyy}' in it somewhere. Sweet - you've got curl working.
Next up we want to write a script to do all the work. I've used ruby, because it's awesome, but you can use whatever you want. The following code opens up the 'pc-full.csv' file, reads the list of distinct postcodes, runs curl against each one, parses the longitude and latitude from each, and outputs a nice CSV file with 3 columns: Postcode, Latitude, Longitude.
Now i'll have to apologise for the sloppy code, but it does work (updated to fix wordpress' curly quotes):
def IsGood(fname)
# Does a file contain the longitude eg did it connect to gmaps correctly?
# Ironically, in C# this code would be a one liner: File.ReadAllText(fname).Contains("center:{lat:");
r = "Missing"
if File.exist?(fname)
f = File.new(fname)
lines = f.read
f.close
if lines.include?("center:{lat:")
r = 'Good'
else
r = 'Bad'
end
end
r
end
# get the list of unique postcodes from the CSV file downloaded from australia post
@postcodes = []
File.new('pc-full.csv').readlines.each {|l|
x = l.gsub(/[\",]/,"").to_i
@postcodes << x if x>0 && !@postcodes.include?(x)
}
puts "Total #{@postcodes.length} unique postcodes"
# Scrape them all from google maps
@postcodes.each {|postcode|
fname = "data_#{postcode}.txt"
if !File.exist?(fname)
system "curl -o #{fname} \"http://maps.google.com.au/" +
"maps?f=q&hl=en&geocode=&q=#{"%04d" % postcode}+australia&output=js\""
puts "Any good? #{IsGood(fname)}"
end
}
# Go through the resultant files, parsing the longitude and latitude
@results = ["Postcode,Lat,Lng"]
@postcodes.each {|postcode|
fname = "data_#{postcode}.txt"
status = IsGood(fname)
puts "#{fname} : #{status}"
# Grab the long & lat
if status=='Good'
f = File.new(fname)
lines = f.read
f.close
m = /center:\{lat:([\-.0-9]*),lng:([\-.0-9]*)\}/.match(lines)
@results << "#{postcode},#{m[1]},#{m[2]}"
end
}
# Write it out to a CSV file
File.open("PostcodeLatLng.csv","w") {|f|
@results.each {|line|
f.write line
f.write "\n"
}
}
Note: Put curl.exe in the same folder as your ruby file above, if it's not in the path.
And your output from running all that should be a 'PostcodeLatLng.csv' file with the contents something like this:
Postcode,Lat,Lng 2000,-33.869027000000003,151.21024499999999 2001,-37.808776999999999,144.94928899999999 2002,-25.335448,135.74507600000001 2004,-33.891787999999998,151.17625100000001
Bob's your uncle! You should now have the longitude and latitude of all australian postcodes. In a later article, i'll show how i use this to make a map of australian post offices overlaying a map of australia, for instance.
The other day i had to figure out how to convert HSV to RGB using C#, and after browsing the net i could only find C sample code to do it with, so i had to convert to C#. The code may be useful to someone out there (please let me know if it is):
/// <summary>
/// Convert HSV to RGB
/// h is from 0-360
/// s,v values are 0-1
/// r,g,b values are 0-255
/// Based upon http://ilab.usc.edu/wiki/index.php/HSV_And_H2SV_Color_Space#HSV_Transformation_C_.2F_C.2B.2B_Code_2
/// </summary>
void HsvToRgb(double h, double S, double V, out int r, out int g, out int b)
{
// ######################################################################
// T. Nathan Mundhenk
// mundhenk@usc.edu
// C/C++ Macro HSV to RGB
double H = h;
while (H < 0) { H += 360; };
while (H >= 360) { H -= 360; };
double R, G, B;
if (V <= 0)
{ R = G = B = 0; }
else if (S <= 0)
{
R = G = B = V;
}
else
{
double hf = H / 60.0;
int i = (int)Math.Floor(hf);
double f = hf - i;
double pv = V * (1 - S);
double qv = V * (1 - S * f);
double tv = V * (1 - S * (1 - f));
switch (i)
{
// Red is the dominant color
case 0:
R = V;
G = tv;
B = pv;
break;
// Green is the dominant color
case 1:
R = qv;
G = V;
B = pv;
break;
case 2:
R = pv;
G = V;
B = tv;
break;
// Blue is the dominant color
case 3:
R = pv;
G = qv;
B = V;
break;
case 4:
R = tv;
G = pv;
B = V;
break;
// Red is the dominant color
case 5:
R = V;
G = pv;
B = qv;
break;
// Just in case we overshoot on our math by a little, we put these here. Since its a switch it won't slow us down at all to put these here.
case 6:
R = V;
G = tv;
B = pv;
break;
case -1:
R = V;
G = pv;
B = qv;
break;
// The color is not defined, we should throw an error.
default:
//LFATAL("i Value error in Pixel conversion, Value is %d", i);
R = G = B = V; // Just pretend its black/white
break;
}
}
r = Clamp((int)(R * 255.0));
g = Clamp((int)(G * 255.0));
b = Clamp((int)(B * 255.0));
}
/// <summary>
/// Clamp a value to 0-255
/// </summary>
int Clamp(int i)
{
if (i < 0) return 0;
if (i > 255) return 255;
return i;
}
And here's how you'd use it:
int r,g,b; HsvToRgb(110, 1, 1, out r, out g, out b);
The TcpClient class in C# is great for opening a TCP connection, I must say that it's one of the nicest TCP libraries i've used. You just have to watch out for the occasional bug and you'll be right.
One limitation is a frustration though: the inability to set a timeout when opening a connection to a remote server. The default timeout is 60 seconds, which is quite a while to have the user strumming their fingers waiting for things to happen.
My solution to this is to spawn a thread which opens the TCP connection, while the original thread waits up to a user-specified timeout for it to connect. If it hasn't connected by then, it kills the thread and gives up. Notice the use of the thread's Join function which allows the original thread to stop waiting if the connection is quicker than the timeout. Without further ado, the TcpClientWithTimeout.cs class:
using System; using System.Net.Sockets; using System.Threading; ////// TcpClientWithTimeout is used to open a TcpClient connection, with a user definable connection timeout in milliseconds (1000=1second) /// Use it like this: /// TcpClient connection = new TcpClientWithTimeout('127.0.0.1',80,1000).Connect(); /// public class TcpClientWithTimeout { protected string _hostname; protected int _port; protected int _timeout_milliseconds; protected TcpClient connection; protected bool connected; protected Exception exception; public TcpClientWithTimeout(string hostname,int port,int timeout_milliseconds) { _hostname = hostname; _port = port; _timeout_milliseconds = timeout_milliseconds; } public TcpClient Connect() { // kick off the thread that tries to connect connected = false; exception = null; Thread thread = new Thread(new ThreadStart(BeginConnect)); thread.Start(); // wait for either the timeout or the thread to finish thread.Join(_timeout_milliseconds); if (connected == true) { // it succeeded, so return the connection thread.Abort(); return connection; } if (exception != null) { // it crashed, so return the exception to the caller thread.Abort(); throw exception; } else { // if it gets here, it timed out, so abort the thread and throw an exception thread.Abort(); string message = string.Format("TcpClient connection to {0}:{1} timed out", _hostname, _port); throw new TimeoutException(message); } } protected void BeginConnect() { try { connection = new TcpClient(_hostname, _port); // record that it succeeded, for the main thread to return to the caller connected = true; } catch (Exception ex) { // record the exception for the main thread to re-throw back to the calling code exception = ex; } } }
And here's a little example of how to use this to open a connection, send 10 bytes, and receive 10 bytes:
// connect with a 5 second timeout on the connection
TcpClient connection = new TcpClientWithTimeout("www.google.com", 80, 5000).Connect();
NetworkStream stream = connection.GetStream();
// Send 10 bytes
byte[] to_send = {0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0xa};
stream.Write(to_send, 0, to_send.Length);
// Receive 10 bytes
byte[] readbuf = new byte[10]; // you must allocate space first
stream.ReadTimeout = 10000; // 10 second timeout on the read
stream.Read(readbuf, 0, 10); // read
// Disconnect nicely
stream.Close(); // workaround for a .net bug: http://support.microsoft.com/kb/821625
connection.Close();
Cheers all, please let me know if this is useful or if you have any constructive criticism.
Oracle Explorer - a very simple C# open source Toad alternative
0 Comments Published July 31st, 2008 in codeHi all, here's Oracle Explorer, a very simple C# (VS2005) open source Toad alternative for browsing through oracle databases.
It was created in reaction to the awful load times of oracle's java-based Sql Developer, and i can't be bothered with the procurement process for Toad. Maybe someone wants to start a project with this and take it further?
Anyway here's the obligatory screenshot:
And the download (including both the EXE and source code): (updated to fix 2 bugs)
Oracle Explorer source and exe
To install and run it, make sure you get the Instant Client from oracle's website, and copy the oci.dll, orannzsbb*.dll, oraocci*.dll, and oraociicus*.dll files into the same folder as the oracle explorer EXE file.
This project is a follow on from a previous blog entry here: Connecting to Oracle from C# / Winforms / Asp.net without tnsnames.ora
Constructive comments are very welcome! (But abuse will be ignored & removed - sheesh!)
Linking DigitalMars' D with a C library (Mongrel's HTTP parser)
0 Comments Published June 23rd, 2008 in codeThis post is intended to give a step-by-step example of linking D with a C library. If you want to mimic my windows development environment, to start with, you'll need to download Tango and DigitalMars C. I used the below versions, but you may also want to check for the latest. I'm also toying with the idea of creating a Rack-style D http server, and coming up with a D web framework. Download dm850c.zip from here, and tango-0.99.6-bin-win32-dmd.1.029.zip from here.
Amusingly, i'd like to highlight that here we're combining Ragel, which creates the Mongrel HTTP parser C code, which was intended for a Ruby web server, finally linking it with D.
Download this article's code here
Make sure you read this first, too.
The (awesome) Mongrel HTTP parser
I grabbed parser.c and parser.h from Ebb, as i had problems with the Mongrel one. You can get it here.
Place the two files into a folder and run the following command to compile them with this command:
\dm\bin\dmc -c parser.c -I.
This will produce the parser.obj if you've installed dmc (Digitalmars' C) correctly.
Bindings between C and D
To make bindings to D, I basically placed the contents of the parser.h into an 'extern(C)' block and deleted all the 'const' references, like so:
extern(C)
{
enum { MONGREL_CONTENT_LENGTH
, MONGREL_CONTENT_TYPE
, MONGREL_FRAGMENT
, MONGREL_HTTP_VERSION
, MONGREL_QUERY_STRING
, MONGREL_REQUEST_PATH
, MONGREL_REQUEST_METHOD
, MONGREL_REQUEST_URI
};
typedef void (*field_cb)(void *data, char *field, size_t flen, char *value, size_t vlen);
typedef void (*element_cb)(void *data, int type, char *at, size_t length);
struct http_parser {
int cs;
int overflow_error;
size_t body_start;
size_t content_length;
size_t nread;
size_t mark;
size_t field_start;
size_t field_len;
size_t query_start;
void *data;
field_cb http_field;
element_cb on_element;
};
void http_parser_init(http_parser *parser);
int http_parser_finish(http_parser *parser);
size_t http_parser_execute(http_parser *parser, char *data, size_t len, size_t off);
int http_parser_has_error(http_parser *parser);
int http_parser_is_finished(http_parser *parser);
}
Since mongrel uses a couple of call-back functions, they must also be within 'extern(C)' blocks:
extern (C) void http_field_cb(void *data, char *field, size_t flen, char *value, size_t vlen)
{ ...code here... }
extern (C) void on_element(void *data, int type, char *at, size_t length)
{ ...code here... }
The D Server
I'm using Tango in the simplest possible way to listen to port 80, feed the input to Mongrel, and return an html page. Here's the entire serve.d:
import tango.io.Stdout;
import tango.stdc.stringz;
import tango.net.ServerSocket, tango.net.SocketConduit;
// Stuff for the mongrel parser
extern(C)
{
enum { MONGREL_CONTENT_LENGTH
, MONGREL_CONTENT_TYPE
, MONGREL_FRAGMENT
, MONGREL_HTTP_VERSION
, MONGREL_QUERY_STRING
, MONGREL_REQUEST_PATH
, MONGREL_REQUEST_METHOD
, MONGREL_REQUEST_URI
};
typedef void (*field_cb)(void *data, char *field, size_t flen, char *value, size_t vlen);
typedef void (*element_cb)(void *data, int type, char *at, size_t length);
struct http_parser {
int cs;
int overflow_error;
size_t body_start;
size_t content_length;
size_t nread;
size_t mark;
size_t field_start;
size_t field_len;
size_t query_start;
void *data;
field_cb http_field;
element_cb on_element;
};
void http_parser_init(http_parser *parser);
int http_parser_finish(http_parser *parser);
size_t http_parser_execute(http_parser *parser, char *data, size_t len, size_t off);
int http_parser_has_error(http_parser *parser);
int http_parser_is_finished(http_parser *parser);
}
// End of stuff for the mongrel parser
// My callback functions
extern (C) void http_field_cb(void *data, char *field, size_t flen, char *value, size_t vlen)
{
char[]* display = cast(char[]*)data;
*display ~= "<i>http_field_cb</i> <b>";
*display ~= field[0..flen];
*display ~= "</b> = ";
*display ~= value[0..vlen];
*display ~= "<br>";
}
extern (C) void on_element(void *data, int type, char *at, size_t length)
{
char[]* display = cast(char[]*)data;
char[] line;
line ~= "<i>on_element</i> <b>";
if (type==MONGREL_CONTENT_LENGTH) line ~= "MONGREL_CONTENT_LENGTH";
if (type==MONGREL_CONTENT_TYPE) line ~= "MONGREL_CONTENT_TYPE";
if (type==MONGREL_FRAGMENT) line ~= "MONGREL_FRAGMENT";
if (type==MONGREL_HTTP_VERSION) line ~= "MONGREL_HTTP_VERSION";
if (type==MONGREL_QUERY_STRING) line ~= "MONGREL_QUERY_STRING";
if (type==MONGREL_REQUEST_PATH) line ~= "MONGREL_REQUEST_PATH";
if (type==MONGREL_REQUEST_METHOD) line ~= "MONGREL_REQUEST_METHOD";
if (type==MONGREL_REQUEST_URI) line ~= "MONGREL_REQUEST_URI";
line ~= "</b> = " ~ at[0..length] ~ "<br>";
*display ~= line;
}
void main()
{
// Set up the server
Stdout("Now open your browser to http://localhost/").newline();
Stdout("Ctrl-Break or Ctrl-C to quit").newline();
auto server = new ServerSocket (new InternetAddress(80));
while(1)
{
// wait for requests
SocketConduit request = server.accept;
// wait for the 'http get ...'
char[1024] response;
uint len = request.read (response);
Stdout(response[0..len]).newline; // you'll want to comment this out if you run apachebench against this...
// parse it
char[] display;
display = "";
http_parser parser;
http_parser_init(&parser);
parser.data = &display; // data that is sent to the callback function
parser.http_field = &http_field_cb;
parser.on_element = &on_element;
http_parser_execute( &parser
, cast(char*)response
, len
, 0
);
display ~= http_parser_has_error(&parser) ? "-mongrel error;" : "-no mongrel error;";
display ~= http_parser_is_finished(&parser) ? "mongrel finished-" : "mongrel not finished-";;
// send HTML
request.output.write("HTTP/1.1 200 OK
<html>
<style>
body {font-family:trebuchet ms;background:#524D4A;color:#fff;}
.comment {font-size:80%;margin:0;color:#cccccc;}
form {background:#6B7552; margin:20px 0 20px 0; padding:5px;}
</style>
<body>
<h1>D + Mongrel's HTTP parser</h1>
<form method='get'>
<h2>GET form</h2>
<input name='blah' value='test value'>
<input type='submit'>
</form>
<form method='post'>
<h2>POST form</h2>
<input name='blah' value='test value'>
<input type='submit'>
</form>
<form method='post'>
<h2>POST form with file</h2>
<input type='file' name='blah'>
<input type='submit'>
</form>
<form>
<h2>Mongrel Parser Results:</h2>
" ~ display ~ "
</form>
</body>
</html>");
request.close();
}
}
Compiling it all together
To compile it together, use the following command, which will produce serve.exe:
\dmd\bin\dmd serve.d parser.obj
Then open your web browser to http://localhost/ and you should see this:
Voila! That's all until next time.
Connecting to Oracle from C# / Winforms / Asp.net without tnsnames.ora
1 Comment Published June 16th, 2008 in codeSummary:
The other day, I needed to access an oracle database from a C# console application, and google found a whole heap of different ways to achieve this. Here's the best way i found that works, and you don't even need to install any oracle software. I've deliberately kept this entry short, to emphasise that this is the simple and straightforward way.
Also, you may want to check out another post for a simple application using this technique: Oracle Explorer - very simple open source Toad alternative
Oracle bits:
Full credit: This part uses details from http://www.codeproject.com/KB/database/C__Instant_Oracle.aspx
You'll need the following files, which can be obtained from within the 'basic lite' version of oracle's instant client:
oci.dll
orannzsbb11.dll
oraocci11.dll
oraociicus11.dll
In my case, i downloaded the 'instantclient-basiclite-win32-11.1.0.6.0.zip' file from their website, the link is here.
You'll need to place these DLL's in the same folder as your application's .EXE. If you're using asp.net, put it in the 'Bin' folder of your application. The important thing to note is that you don't need to install the instant client, or register these DLLs or anything, just grab them from the zip file and ignore the rest of its contents.
Visual Studio bits:
Note that i'm using .Net 2, with Visual Studio 2005. There may be differences in other versions, however i'm hoping for your sake that it is similar enough to figure out.
In the solution explorer, right click on 'References' and choose 'Add Reference…' as below:

Then scroll through the list until you find 'System.Data.OracleClient' and click OK.

Code:
Now for the code. You'll need this 'using' at the top of your file:
using System.Data.OracleClient;
Here's a simple function I wrote to create an oracle connection string, it may be useful to you:
public string OracleConnString(string host,string port,string servicename,string user,string pass)
{
return String.Format(
"Data Source=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST={0})" +
"(PORT={1}))(CONNECT_DATA=(SERVICE_NAME={2})));User Id={3};Password={4};",
host,
port,
servicename,
user,
pass);
}
And here's the gist of opening a connection, and reading all the lines. You'll notice the interface is pretty much the same as accessing SQL server.
string connectionstring = OracleConnString("aaa","1521","bbb","ccc","ddd");
string sql = "select * from some_table";
using (OracleConnection conn = new OracleConnection(connectionstring)) // connect to oracle
{
conn.Open(); // open the oracle connection
using (OracleCommand comm = new OracleCommand(sql, conn)) // create the oracle sql command
{
using (OracleDataReader rdr = comm.ExecuteReader()) // execute the oracle sql and start reading it
{
while (rdr.Read()) // loop through each row from oracle
{
Console.WriteLine( rdr[0] ); // You can do this
Console.WriteLine( rdr.GetString(0); ); // or this
Console.WriteLine( rdr["column_name"] ); // or this
}
rdr.Close(); // close the oracle reader
}
}
conn.Close(); // close the oracle connection
}
That's all!
Copying from Oracle to Sql Server
Here's an example of copying a 2-column table from Oracle to Sql server. Lets pretend this table is called 'foo', and has two integer columns 'x' and 'y'.
private void CopyOracleToSql()
{
string connectionstring = OracleConnString("host","1521","servicename","user","pass");
using (OracleConnection conn = new OracleConnection(connectionstring)) // connect to oracle
{
conn.Open(); // open the oracle connection
string sql = "select x,y from foo";
using (OracleCommand comm = new OracleCommand(sql, conn)) // create the oracle sql command
{
using (OracleDataReader rdr = comm.ExecuteReader()) // execute the oracle sql and start reading it
{
using (SqlConnection sql_conn = new SqlConnection(sqlconnstring)) // connect to the sql 2005 database
{
sql_conn.Open(); // open the sql database
// Read the data from oracle, and every 100 rows send it to the sql database
StringBuilder sb = new StringBuilder(); // build the insert statements with this
int row = 0;
while (rdr.Read()) // loop through each row from oracle
{
sb.AppendFormat(
"insert into foo(x,y) values({0},{1});",
rdr[0], rdr[1]);
row++;
if (row >= 100) // send to the DB every 100 rows
{
using (SqlCommand sql_comm = new SqlCommand(sb.ToString(), sql_conn))
{
sql_comm.ExecuteNonQuery();
row = 0;
sb = new StringBuilder();
}
}
}
// get the remaining few rows and send them to the DB
if (sb.Length > 0)
{
using (SqlCommand sql_comm = new SqlCommand(sb.ToString(), sql_conn))
{
sql_comm.ExecuteNonQuery();
}
}
rdr.Close(); // close the oracle reader
sql_conn.Close(); // close the sql connection
}
}
}
conn.Close(); // close the oracle connection
}
}
Firstly, a note: The aim of this article is more of a step by step 'first steps with D' kinda thing, i'm not trying to make too much of a point about D's performance here. Calm down, redditors!
Environment
I'm working with Windows XP here, which adds a few complications you wouldn't see on other platforms.
Using the Digitalmars' C compiler (dm850c.zip) from here, unzipped to c:\dm.
Also using the Tango D package (tango-0.99.6-bin-win32-dmd.1.029.zip) from here, unzipped to c:\dmd.
Libev
Libev is a high performance event loop that i'm using to wait for incoming port 80 connections.
It's here: http://software.schmorp.de/pkg/libev.html
Docs are here: http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod
Download the CVS version (it contains fixes so it'll compile) here: http://cvs.schmorp.de/libev.tar.gz?view=tar
Unzip it all into a folder, and create a 'myev.c' file in that folder, with the following contents:
#define EV_STANDALONE 1 // so i don't need a config.h file
#define EV_SELECT_IS_WINSOCKET 1 // windows
//#define EV_USE_POLL 1 // uncomment me for unix
//#define EV_USE_EPOLL 1 // uncomment me for gnu/linux
//#define EV_USE_KQUEUE 1 // uncomment me for bsd/osx
//#define EV_USE_PORT 1 // uncomment me for solaris
#define EV_STAT_ENABLE 0 // disable file watching
#include "ev.c"
To compile it, do the following from that folder (you'll get a couple of warnings which you can ignore):
\dm\bin\dmc myev.c -c
This will produce a myev.obj which you'll need later
Server
1: Create a new folder for the simple 'D' server. You'll need the ev.d with the libev bindings, which you can download or copy n paste from here.
2: Copy the 'myev.obj' that you compiled earlier from the libev folder into this folder.
3: You'll then create the 'server.d' which is the main point of this article:
import tango.io.Console;
import tango.net.ServerSocket, tango.net.SocketConduit;
import ev;
extern (C)
{
static void libev_cb (ev_loop_t *loop, ev_io *w, int revents)
{
callback();
}
// http://msdn.microsoft.com/en-us/library/bdts1c9x(VS.71).aspx
int _open_osfhandle (long osfhandle, int flags);
}
ServerSocket listener;
void main()
{
listener = new ServerSocket (new InternetAddress(80));
int fd = _open_osfhandle(listener.fileHandle,0); // for win32: convert from socket to file descriptor
// Start libev
ev_loop_t* loop = ev_default_loop(0);
ev_io io_watcher;
ev_io_init(&io_watcher, &libev_cb, fd, READ);
ev_io_start(loop, &io_watcher);
ev_loop(loop, 0);
}
void callback()
{
// accept the connection
SocketConduit request = listener.accept;
// wait for the 'http get ...'
char[1024] response;
uint len = request.input.read (response);
Cout (response[0..len]).newline; // you'll want to comment this out if you run apachebench against this...
// send HTML
request.output.write("HTTP/1.1 200 OK
<html>
<style>body {font-family:trebuchet ms;background:#424242;color:#fff;text-align:center;margin-top:10em;}</style>
<body><h1>Your 'D' web server is alive!</h1></body>
</html>");
request.close();
}
To compile and run all this, do the following:
\dmd\bin\dmd server.d ev.d myev.obj
server
Next test it in your browser, by browsing to 'localhost':
Voila! And if i do an apachebench (ab -n 10000 -c 100), my requests-per-second is 2071 for an 2 years old P4-3ghz, and it only uses 2.4mb of memory, which is pretty impressive considering each Ruby on Rails Mongrel typically uses ~30megs last time i checked.
I guess the next question is: why doesn't someone make a super-scalable web framework using the speed/efficiency and ease of D?
If any of this doesn't work, make sure you post a comment so i can sort it out - cheers.
I've recently been working on converting my 'rosters' rails 1.x app to rails version 2, and there are plenty of pitfalls that i spent lots of time googling around trying to find the solutions.
So, for just as much need for my own future reference as anyone else's, here are all the specific traps *I* ran into, and their solutions, in the one place.
If you're interested in the source code to the whole app, you can see the rails 1.x version here:
http://rosters.rubyforge.org/
The new version is running here, and if enough people request it, i'll release the code.
Basics
Firstly, I created a new rails 2 skeleton app, using 'rails myappname' from the command line. Then i copied the contents of all my app and public folders over from the old app to the new one. Finally i went through the db and config directories, modifying the new files manually, rather than simply overwriting them with the old ones. This way you'll get to see how the configuration files (especially routes!) have changed, and won't be overwriting them with legacy code (it feels strange to refer to rails 1 as 'legacy'!)
What? Rails is embracing security now?
Forms now have a special 'key' that needs to be present for POSTs, to prevent cross-site-request-forgery (try saying that 10 times fast!). The gist of it is that you can't just have code like this in your views any more:
<form name="f" action="/account/login" method="post" onsubmit="wait();">
...
</form>
Now what you want is more like this:
<% form_tag("/account/login", :name=>'f', :onsubmit=>"wait();") { %>
...
<% } %>
The above code takes care of the new CSRF-proof keys and all that. As you can see, my example is a bit more complex than usual, with a form name and an onsubmit javascript callback, but hey, its real code!
For a simpler, bare-bones form, Rails 2 style, here you go:
<% form_tag :action=> "new" do %>
...
<% end %>
Where on earth did 'find_first' go?
Back in controller-land, i hit problems with all my find_first's that i like to use. This kind of thing simply wont work any more:
u = find_first(["login = ? AND password = ?", login, sha1(pass)])
You need something more like this nowadays:
u = find :first, :conditions=>["login = ? AND password = ?", login, sha1(pass)]
It appears that all the find_* functions (find_first,find_all,etc) have all been rolled into the one-function-to-rule-them-all 'find' function. I guess it is neater.
No more @session or @request
This one's simple. Simply replace these kind of things:
@session
@request
With this:
session
request
One big 'find-in-files' (Ctrl-Shift-F in Notepad++) will sort you out with this one.
Redirect
Another simple one, but its likely you'll have one of these in each of your updating actions:
redirect_to_url '/blah/foo'
...becomes...
redirect_to '/yada/yada'
RIP Pagination
Apparently pagination wasn't quite good enough to meet muster for Rails 2, so its been scrapped totally.
There's no simple replacement for this one, but hopefully you're only really using it in your admin scaffolds, so what i did was a simple pagination-dectomy in my controllers:
@user_pages, @users = paginate :users, :per_page => 30, :order=>'login'
Became:
@users = User.find :all, :order=>'login'
Its pretty simple, and you'll need to remove all references to pagination in your views. If you really *really* need pagination, you'll need something more complex, but there isn't anything out-of-the-box for you.
For more details: http://wiki.rubyonrails.org/rails/pages/HowtoPagination
End Results
Well, now the site's up and running, you're all welcome to have a look:
http://rosters.morphexchange.com/
DanceInforma is a web-zine devoted to dancers (hence the name).
My role consists of maintaining the back end database involved with maintaining the membership details of all members, ~3-4000 at last count.
http://danceinforma.com/
FreelanceSwitch contracted me to develop their Client Analyser, which is publicly accessible here:
http://clientanalyser.freelanceswitch.com/
This is another Rails application, like most of my public work.


