X-Git-Url: http://git.bytex64.net/?a=blobdiff_plain;f=www%2Fdoc%2Findex.html;h=25f6519f08cbce3d88ef46bdcf7d06a1572cb4d3;hb=HEAD;hp=9b2e3dfb481bd0a44885245a26d4184483b80faa;hpb=46199d2fb2fdbfad11b3044bddaa817270c2f44f;p=blerg.git diff --git a/www/doc/index.html b/www/doc/index.html index 9b2e3df..25f6519 100644 --- a/www/doc/index.html +++ b/www/doc/index.html @@ -3,6 +3,7 @@ Blërg Documentation + @@ -17,7 +18,7 @@ as either a standalone HTTP server, or a CGI. Blërg is written in pure C. -

Installing

+

Running Blërg

Getting the source

@@ -62,7 +75,7 @@ http://git.bytex64.net/blerg.git. — as a standalone HTTP server, or as a CGI. You will need: @@ -81,10 +94,8 @@ sense of humor, requires ruby to compile)

Configuring

-

I know I'm gonna get shit for not using an autoconf-based system, but -I really didn't want to spend time figuring it out. You should edit -libs.mk and put in the paths where you can find headers and libraries -for the above requirements. +

Edit libs.mk and put in the paths where you can find headers and +libraries for the above requirements.

Also, further apologies to BSD folks — I've probably committed several unconscious Linux-isms. It would not surprise me if the @@ -95,11 +106,11 @@ more portable, I'd be happy to hear them.

Building

At this point, it should be gravy. Type 'make' and in a few seconds, -you should have blerg.httpd, blerg.cgi, -rss.cgi, and blergtool. Each of those can be -made individually as well, if you, for example, don't want to install -the prerequisites for blerg.httpd or -blerg.cgi. +you should have blerg.cgi, blergtool, and +blerglatest. + +

NOTE: blerg.httpd is deprecated and will not be +updated with new features.

Installing

@@ -109,26 +120,17 @@ reason, it's better to use a subdomain (i.e., blerg.yoursite.com is easier than yoursite.com/blerg/). If you do want to put it in a subdirectory, you will have to modify www/js/blerg.js and change baseURL at the top as well as a number of other self-references -in that file and www/index.html. The CGI version should -work fine this way, but the HTTP version will require the request to be -rewritten, as it expects to be serving from the root. +in that file and www/index.html.

You cannot serve the database and client from different domains (i.e., yoursite.com vs othersite.net, or even foo.yoursite.com and bar.yoursite.com). This is a requirement of the web browser — the same origin policy will not allow an AJAX request to travel across -domains. - -

For the standalone web server:

- -

Right now, blerg.httpd doesn't serve any static assets, -so you're going to have to put it behind a real webserver like apache, -lighttpd, nginx, or similar. Set the document root to the www -directory, then proxy /info, /create, /login, /logout, /get, /tag, and -/put to blerg.httpd. You can change the port blerg.httpd -listens on in config.h. +domains (though you can probably get around it these days with Cross-origin + resource sharing). -

For the CGI version:

+

For straight CGI with Apache

Copy the files in www/ to the root of your web server. Copy blerg.cgi to your web server. Included in www-configs/ is @@ -136,10 +138,47 @@ a .htaccess file for Apache that will rewrite the URLs. If you need to call the CGI something other than blerg.cgi, the .htaccess file will need to be modified. +

For nginx

+ +

Nginx can't run CGI directly, and there's currently no FastCGI +version of Blërg, so you will have to run it under some kind of CGI to +FastCGI gateway, like the one described here on the nginx wiki. This +pretty much destroys the performance of Blërg, but it's all we've got +right now. +

The extra RSS CGI

-

There is an optional RSS cgi (rss.cgi) that will serve +

There is an optional RSS cgi (aux/cgi/rss.cgi) that will serve RSS feeds for users. Install this like blerg.cgi above. +As of 1.9.0, this is a perl FastCGI script, so you will have to make +sure the perl libraries are available to it. A good way of doing that +is to install to an environment directory, as described below. + +

Installing to an environment directory

+ +

The Makefile has support for installing Blërg into a directory that +includes tools, libraries, and configuration snippets for shell and web +servers. Use it as make install-environment + ENV_DIR=<directory>. Under <directory>/etc will be +a shell script that sets environment variables, and configuration +snippets for nginx and apache to do the same. This should make it +somewhat easier to use Blërg in a self-contained way. + +

For example, this will install Blërg to an environment directory +inside your home directory: + +

user@devhost:~/blerg$ make install-environment ENV_DIR=$HOME/blerg-env
+...
+user@devhost:~/blerg$ . ~/blerg-env/etc/env.sh
+
+ +

Then, you will be able to run tools like blergtool, and +it will operate on data inside ~/blerg-env/data. Likewise, +you can include +/home/user/blerg-env/etc/nginx-fastcgi-vars.conf or +/home/user/blerg-env/etc/apache-setenv.conf in your +webserver to make the CGI/FastCGI scripts to the same thing.

API

@@ -156,7 +195,7 @@ root of the wesite. response, like 404 Not Found if a record or user doesn't exist, or a 200 response with a 'JSON failure', which will look like this: -

{"status": "failure"} +

{"status": "failure"}

Blërg doesn't currently explain why there is a failure, and I'm not sure it ever will. @@ -165,7 +204,7 @@ I'm not sure it ever will. /get, /tag, or /info), or a 'JSON success' response (for /create, /put, /login, or /logout), which looks like this: -

{"status": "success"} +

{"status": "success"}

For the CGI backend, you may get a 500 error if something goes wrong. For the HTTP backend, you'll get nothing (since it will have crashed), @@ -180,6 +219,20 @@ because I use strncmp(3) to compare).

Tags must be 64 characters or less, and can contain only the ASCII characters 0-9, A-Z, a-z, underscore (_), and hyphen (-). +

Authorization

+ +

As the result of a successful login, the server +will send back a cookie named auth. This cookie authorizes +restricted requests, and must be sent for any API endpoint marked authorization, or else you will get a 403 Forbidden +response. The cookie format looks like: + +auth=username/abcdef0123456789abcdef0123456789 + +That is a username, a forward slash, and 32 hexadecimal digits which denote the +"token" identifying the session. On logout, the server will invalidate the +token and expire the cookie. +

/create - create a new user

To create a user, POST to /create with username and @@ -195,24 +248,24 @@ success if the user is created. respond with JSON failure if the user does not exist or if the password is incorrect. On success, the server will respond with JSON success, and will set a cookie named 'auth' that must be sent by the client when -accessing restricted API functions (/put and /logout). +accessing restricted API functions (See Authorization above).

/logout - log out

+
authorization
-

POST to /logout with with username, the user to log out, -along with the auth cookie in a Cookie header. The server will respond -with JSON failure if the user does not exist or if the auth cookie is -bad. The server will respond with JSON success after the user is -successfully logged out. +

POST to /logout. The server will respond with JSON failure if the +user does not exist or if the request is unauthorized. The server will +respond with JSON success after the user is successfully logged out.

/put - add a new record

+
authorization
-

POST to /put with username and data -parameters, and an auth cookie. The server will respond with JSON -failure if the auth cookie is bad, if the user doesn't exist, or if -data contains more than 65535 bytes after URL -decoding. The server will respond with JSON success after the record is -successfully added. +

POST to /put with a data parameter. The server will +respond with JSON failure if the request is unauthorized, if the user +doesn't exist, or if data contains more than 65535 bytes +after URL decoding. The server will respond with JSON success +after the record is successfully added.

/get/(user), /get/(user)/(start record)-(end record) - get records for a user

@@ -281,6 +334,99 @@ extra author field, like so:

There is currently no support for getting more than 50 tags, but /tag will probably mutate to work like /get. +

/subscribe/(user) - Subscribe to a +user's updates

+
authorization
+ +

POST to /subscribe/(user) with a subscribed parameter +that is either "true" or "false", indicating whether (user) should be +subscribed to or not. The server will respond with JSON failure if the +request is unauthorized or if the user doesn't exist. The server will +respond with JSON success after the subscription request is successfully +registered. + +

/feed - Get updates for subscribed users

+
authorization
+ +

POST to /feed, with a username parameter and an auth +cookie. The server will respond with a JSON list of the last 50 updates +from all subscribed users, in reverse chronological order. Fetching +/feed does not reset the new message count returned from /status. To do +that, look at POST /status. + +

NOTE: subscription notifications are only stored while subscriptions +are active. Any records inserted before or after a subscription is +active will not show up in /feed. + +

/status, /status/(user) - Get or clear +general and user-specific status

+
authorization
+ +

GET to /status to get information about your account. It tells you +the number of new subscription records since the last time the +subscription counter was reset, and a flag for whether the account was +mentioned since the last time the mention flag was cleared. The server +will respond with a JSON object: + +

+{
+  "feed_new": 3,
+  "mentioned": false
+}
+
+ +

POST to /status with a clear parameter that is either +"feed" or "mentioned" to reset either the subscription counter or the +mention flag, respectively. There is not currently a way to clear both +with a single request. The server will respond with JSON success. + +

GET to /status/(user) to get subscription information for a +particular user. The server will respond with a simple JSON object: + +

+{"subscribed":true}
+
+ +

The value of "subscribed" will be either true or false depending on +the subscription status. + +

/passwd - Change a user's password

+
authorization
+ +

POST to /passwd with password and +new_password parameters to change the user's password. For +extra protection, changing a password requires sending the user's +current password in the password parameter. If +authentication is successful and the password matches, the user's +password is set to new_password and the server responds +with JSON success. + +If the password doesn't match, or one of password or +new_password are missing, the server returns JSON failure. + +

Libraries

+ +

C

+ +

Most of Blërg's core functionality is packaged in a static library +called blerg.a. It's not designed to be public or +installed with `make install-environment`, but it should be relatively +straightforward to use it in C programs. Look at the headers under the +database directory. + +

A secondary library called blerg_auth.a handles the +authentication layer of Blërg. To use it, look at +common/auth.h. + +

Perl

+ +

As of 1.9.0, Blërg includes a perl library called +Blerg::Database. It wraps the core and authentication +functionality in a perlish interface. The module has its own POD +documentation, which you can read with your favorite POD reader, from +the manual installed in an environment directory, or in HTML here. +

Design

Motivation

@@ -338,14 +484,15 @@ make the layers more efficient, or reduce the number of layers.

Blërg does both by smashing the last two or three layers into one -application. Blërg can be run as either a standalone web server, or as -a CGI (FastCGI support is planned, but I just don't care right now). -Less waste, more throughput. As a consequence of this, the entirety of -the application logic that the user sees is implemented in the client -app in Javascript. That's why all the URLs have #'s — the page is -loaded once and switched on the fly to show different views, further -reducing load on the server. Even parsing hash tags and URLs are done -in client JS. +application. Blërg can be run as either a standalone web server +(currently deprecated because maintaining two versions is hard), or as a +CGI (FastCGI support is planned, but I just don't care right now). Less +waste, more throughput. As a consequence of this, the entirety of the +application logic that the user sees is implemented in the client app in +Javascript. That's why all the URLs have #'s — the page is loaded +once and switched on the fly to show different views, further reducing +load on the server. Even parsing hash tags and URLs are done in client +JS.

The API is simple and pragmatic. It's not entirely RESTful, but is rather designed to work well with web-based front-ends. Client data is @@ -364,24 +511,24 @@ early in the design process that I'd try out mmaped I/O. Each user in Blërg has their own database, which consists of a metdata file, and one or more data and index files. The data and index files are memory mapped, which hopefully makes things more efficient by letting the OS -handle when to read from disk (or maybe not &mdash I haven't benchmarked -it). The index files are preallocated because I believe it's more -efficient than writing to it 40 bytes at a time as records are added. -The database's limits are reasonable: +handle when to read from disk (or maybe not — I haven't +benchmarked it). The index files are preallocated because I believe +it's more efficient than writing to it 40 bytes at a time as records are +added. The database's limits are reasonable: - +
maximum record size65535 bytes
maximum number of records per database264 - 1 bytes
maximum number of records per database264 - 1
maximum number of tags per record1024

So as not to create grossly huge and unwieldy data files, the database layer splits data and index files into many "segments" -containing at most 64K entries each. Those of you doing some quick math -in your heads may note that this could cause a problem on 32-bit -machines — if a full segment contains entries of the maximum -length, you'll have to mmap 4GB (32-bit Linux gives each process only -3GB of virtual address space). Right now, 32-bit users should change +containing at most 64K entries each. Those of you doing some quick +mental math may note that this could cause a problem on 32-bit machines +— if a full segment contains entries of the maximum length, you'll +have to mmap 4GB (32-bit Linux gives each process only 3GB of virtual +address space). Right now, 32-bit users should change RECORDS_PER_SEGMENT in config.h to something lower like 32768. In the future, I might do something smart like not mmaping the whole fracking file. @@ -425,6 +572,42 @@ disk before returning success. This should make Blërg extremely fast, and totally unreliable in a crash. But that's the way you want it, right? :] +

Subscriptions

+ +

When I first started thinking about the idea of subscriptions, I +immediately came up with the naïve solution: keep a list of users to +which users are subscribed, then when you want to get updates, iterate +over the list and find the last entries for each user. And that would +work, but it's kind of costly in terms of disk I/O. I have to visit +each user in the list, retrieve their last few entries, and store them +somewhere else to be sorted later. And worse, that computation has to +be done every time a user checks their feed. As the number of users and +subscriptions grows, that will become a problem. + +

So instead, I thought about it the other way around. Instead of doing +all the work when the request is received, Blërg tries to do as much as +possible by "pushing" updates to subscribed users. You can think of it +kind of like a mail system. When a user posts new content, a +notification is "sent" out to each of that user's subscribers. Later, +when the subscribers want to see what's new, they simply check their +mailbox. Checking your mailbox is usually a lot more efficient than +going around and checking everyone's records yourself, even with the +overhead of the "mailman." + +

The "mailbox" is a subscription index, which is identical to a tag +index, but is a per-user construct. When a user posts a new record, a +subscription index record is written for every subscriber. It's a +similar amount of I/O as the naïve version above, but the important +difference is that it's only done once. Retrieving records for accounts +you're subscribed to is then as simple as reading your subscription +index and reading the associated records. This is hopefully less I/O +than the naïve version, since you're reading, at most, as many accounts +as you have records in the last N entries of your subscription index, +instead of all of them. And as an added bonus, since subscription index +records are added as posts are created, the subscription index is +automatically sorted by time! To support this "mail" architecture, we +also keep a list of subscribers and subscrib...ees in each account. +

Problems, Caveats, and Future Work

Blërg probably doesn't actually work like Twitter because I've never