Norbert Hartl

mostly brackets and pipes

James Foster on Maintenance Gem Overhead

If you are using GemStone you know there are a lot of things to tweak to get the best out of it. James Foster has a new blog post about how to get rid of overhead of the maintenance gem. In my installations I’m using a similar approach. Definitely a read

TaskForUs - Gemeinsam Tasks Verwalten

Seit einigen Tagen ist unsere neueste Applikation im AppStore erhältlich. Es gibt wahrscheinlich wenig App Kategorien im Apple AppStore, die so dicht besiedelt sind, wie die der todo Applikationen. Das ist verständlich, wenn man sich Programmiertutorials anschaut, denn dort ist eine todo Applikation ein einfaches Beispiel, das man als Anfänger umsetzen kann. Das Daten-Model ist meist sehr simpel und ein Endergebnis nicht viel mächtiger. Aus dieser Sicht heraus ist es entweder sehr töricht oder mutig, noch eine todo Applikation zu erstellen.

Wir sehen es lieber als letzteres. Unterteilt man die todo Applikationen in ihre Möglichkeiten wie Online-Modus, Backup oder Kollaborationsmöglichkeiten, dann trennt sich die Spreu vom Weizen. Gefühlt sind über 90% der Applikationen, simple Apps, die nur offline todo Einträge anlegen und abhaken können. Der Rest teilt sich in “getting things done” Apps und ich möchte sie komplexe todo Apps nennen. Die komplexen todo Apps bieten eine Fülle von Kategorien und Einstellungen, die einen meist nur mehr verwirren als dass sie helfen würden.

Wenn wir die offline Apps und die komplexen wegnehmen, bleibt eine überschaubare Menge an Apps über. Von diesen bieten die meisten irgendeine Online-Möglichkeit. Sei es das Backup der Daten oder die Verknüpfung mit einem Webinterface oder Desktop-Applikation. Was aber weitgehendst fehlte, als wir begannen unsere Applikation umzusetzen, war die Möglichkeit, Listen mit anderen Personen zu teilen. Wir sind sehr an Kollaborationsmöglichkeiten interessiert, deshalb war dort eine Lücke zu sehen. Um ehrlich zu sein, muss man sagen, dass sich dieser Zustand inzwischen leicht verändert hat. Einige der Apps bieten inzwischen die Möglichkeit, Listen zu teilen. Aber nur wenige.

Die geteilten Listen war unser Hauptfokus bei der Erstellung dieser App. Wir wollten eine simple App haben, die einen möglichst wenig verwirrt, die aber durch ausgewählte Features eine gewisse Mächtigkeit gewinnt, und damit nützlich ist. Das Teilen von Listen macht die Applikation unserer Meinung nach sehr gut. Man erstellt eine Liste und kann dann beliebig viele Personen einladen, die Liste zu teilen. Sobald Daten zwischen mehreren Personen geteilt werden, betritt man einen ganz neuen Komplexen an schwierigen Problemen. Spätestens an diesem Punkt hört die todo App auf, trivial zu sein. Die App sollte jederzeit in der Lage sein, die Inhalte von mehreren Personen zu vereinen. Und diese Änderungen sollten verzögert anwendbar sein. Weniger komplizert ausgedückt bedeutet das, daß ich Aufgaben erstellen möchte, wenn ich offline bin oder kein Netz habe. Und wenn ich dann wieder online bin, möchte ich weder, dass jemand meine Inhalte überschrieben hat, noch daß die App es ablehnt, meine Inhalte anzunehmen.

Wir denken, dass TaskForUs das Problem sehr gut löst und dabei das Datenaufkommen in die cloud schmal hält. Das ist wichtig, weil wir auch unsere Aufgaben abgleichen wollen, wenn wir im Zug sitzen und/oder wir uns in einem edge Netzwerk befinden. Wir hoffen, daß unsere Überlegungen und Umsetzungen sich mit den Wünschen von Leuten decken, die sich mittels Aufgabenlisten organisieren oder es wollen. Wir sind sehr an Feedback interessiert. Auf der Website gibt es genügend Möglichkeiten, mit uns in Kontakt zu treten. Bei wem ich nun etwas Interesse geweckt habe, der schaut sich bitee TaskForUs an

Widerstand Ist Zwecklos?

Mit wenig Mühe habe ich es all die Jahre geschafft, Online-Services wie twitter fernzubleiben. Ich bin davon ausgegangen, alle wüssten, dass Dinge wie facebook und twitter Kreationen der grauen Männer sind, doch kaum einer es schafft zu widerstehen. Ich wollte!Heute war es dann soweit und der Widerstand gebrochen. Vielleicht bedeutet das wenigstens, daß die Tage twitters gezählt sind. Wer weiss!Here we go: @NorbertHartl

Nächste Cologne.js Am 12. Juli 2011

Das nächste Treffen der Kölner javascript user group cologne.js findet am kommenden Dienstag, den 12. Juli 2011 wie gewohnt im CoWoCo in Deutz statt. Wer Interesse hat, der kommt einfach vorbei. Informationen zur Veranstaltung und Vorträgen findet ihr hier

Tidy Up Your Downloads Folder

Downloading stuff from the internet is just something I do every day. On Mac OS it is also quite natural that the files are put into the Downloads folder in your home folder which you can access through the application bar. The newest files are seen at first which makes accessing them really easy. Most of the time I open the files only once and then I forget them.

This works good as long as your disk space is not running short and you are not trying to find an important file you’ve downloaded a week ago but can’t remember the name. Opening the Downloads folder shows you the whole mess. Finding a file is not quite easy in those thousands and thousands of files. While it is hard to find things it is even harder to delete stuff. The files in there might be two seconds old or 1 year. I don’t want to delete really new files because I might need them the other day. I always struggle to tidy up properly. It is really cumbersome and most of the time it gets on my nerves. Then I delete all files in the folder hoping not to loose important stuff.

After all solving the problem was really easy. Files that are really important are saved immediately to the folder they belong to. Files that are rather new should stay in the Downloads folder they belong there. In between are all the candidates for removal. It just needs a rule of what is not new and not important. I think a file that I’ve downloaded and that I didn’t touch within the last three month is not that important. A one liner of shell script makes the rule come true. If you open a terminal on your Mac you just need to paste the next line in the terminal and press enter:

find ~/Downloads -atime +90d -exec mv {} ~/.Trash \;

It finds all files in the Downloads folder that have not been accessed within the last 90 days. Every file that was last touched more than 90 days ago is moved into the trash. In the trash you can let reside them for another period. Or you do quick scan over the files to find files that shouldn’t be deleted. Or you you just empty the trash making room for more removals

Maintaining GemStone Disk Space

It is very easy to start development with GemStone. You can use the virtual appliance provided or you can do a manual installation. In both cases all necessary things are preconfigured so you don’t have to care about the details of GemStone. And GemStone is a smalltalk image and a database which we find way cool while working with it. Persistence is just there, yeah!

Being a database

But this database thingy is not a magic thing after all. It is down-to-earth technology that works beneath. You need to learn about it if you try to do a more serious approach in gemstone maintenance. There are some flavours of how users approach the nuts & bolts of the “being a database”. Some users detect that there are files growing on the disk eating up a lot of disk space. Or they discover through GemTools that their image is growing all the time and they don’t know why. I learned it the hard way back then as GemStone had a size limit on the community edition.

It just stopped working as soon as it reached the limit. Why is that? We said GemStone is a database. A database takes your data and writes it to disk (or something similar). A fault tolerant database uses transaction logs to minimize the possibilities of data corruption. A transaction log is sequential stream of data changes that are written first to disk before the database content itself is changed. So you have two places where the data is written. If an error occurs between the write of the transaction log and the write of the database content this is not a problem. As soon as the database is restarted it detects that the transaction log has newer entries than the databases last change. It can then replay all the transactions that are missing in the database. That’s why they are sometimes called replay logs.

The same thing you can find in filesystems. But then it is called a journaled filesystem. The journal is something like a transaction log. The database and tranlog files in a normal Glass installation you can find in

/opt/gemstone/product/seaside/data 

and it looks like:

-rw-r--r-- 1 gemstone gemstone  520093696 2011-06-03 14:55 extent0.dbf
-rw-r--r-- 1 gemstone gemstone  47648256 2011-06-03 14:55 tranlog2.dbf

Understanding growth

The watchful reader already saw there is a double growth. The image grows which is reflected in the size of the database file (usually extent0.dbf). The database/image is dependent on the number of objects in the image. If new objects are created and persisted the image/database file grows. Note: the database/image file will not shrink if objects are deleted. The space is freed within the database/image file leaving the file on the filesystem at the same size. If new objects are created it will not grow until the freed space in the file has been filled. And transaction logs grow. Transaction logs grow when data is changed. Tranlogs are written sequentially. Each tranlog grows in size until a limit is reached. If the limit is reached a new tranlog file is created and used. The files are sequentially numbered. If a tranlog file with name of tranlog2.dbf is active the next file will be tranlog3.dbf. The maximum size of the tranlog file is a configurable value.

Keeping the image from growing

In case you are using Glass I assume you are using seaside. Seaside sessions are not small. And seaside does not automatically cleanup old sessions for you. You need to do it yourself. If your image is growing steadily than this could be one reason why. After having cleaned the seaside sessions the image is still growing. By cleaning seaside sessions we just detached the objects from the active object graph but they still exist in the image. GemStone does not automatically run the garbage collector to collect these objects and free the space. Have a look at the script in

/opt/gemstone/product/seaside/bin/startMaintenance

This script will run forever and will become active every 60 seconds. It will then remove old seaside sessions and after that it runs a garbage collection. Well, it does a so called “mark for collection” that means it detects all detached objects and adds them to a list of “candidates to remove”. It can take even a few minutes before GemStone starts to actually remove the objects. As the database/image file is not shrinking you won’t see it while watching at the file.

The best option might be to use the option “File Size Report” in GemTools. If you are logged in you can see a button “Admin…”. Pressing the button will open a menu where you can find an item “DoIt…”. Pressing it will open another menu which contains the “File Size Report”. It looks like this:

Extent #1
-----------   
   Filename = !TCP@localhost#dir:/opt/gemstone/log#dbf!/opt/application/myapp/data/extent0.dbf   
   File size =       496.00 Megabytes   
   Space available = 333.22 Megabytes

Here we can see the file is 496 MB big but inside there are more than 333 MB free. So your actual image size is somewhat around 163 MB in size. If the maintenance script is running and your own model is cleaned up as well than you should see only little growth on the database/image file. Or better the growth should correspond to the growth of your domain model.

Maintaining tranlog files

The database/image is now stable in size but the tranlog files are growing further. Even if you just have a small website you can grow tranlog files of more than 1GB a day. If traffic raises or you have multiple stones with multiple sites running it adds up pretty quick to a noticeable amount of space. It might not be a problem. You can just let it run and clean up the space once a month or once a week. Disk space is cheap so you probably don’t need to care.

I need to. We have a dedicated server at a big hoster. The machine has 1TB of disk space but backup space is only 100GB and there is no option to increase it. We have a lot of domains, sites and email accounts. Doing a triple full backup with incremental backups in between fills the space pretty quick. Enough reason to clean up unnecessary garbage. And the tranlogs are of this sort.

Minimizing disk waste

I wrote a script to help with maintaing tranlog files. The script is available in the stone-creator package in the bin/ directory. The script is called delete-old-tranlogs.sh

The script detects old tranlog files that aren’t needed anymore (by investigating the extent0.dbf) file. It can be used to automatically remove those files. It has very few options:

usage: delete-old-tranlogs.sh 
   -d [directory] 
   -g [gemstonedir] 
   -r   
   -d [directory]   data directory of stone (containing extent0.dbf)   
   -g [gemstonedir] directory of gemstone installation                     (default: /opt/gemstone/product)   
   -r               really remove tranlogs. Without this switch they are                    
                    only shown

This will keep old tranlogs from laying around unused. The default maximum size of tranlog file is 1GB. To minimize the disk usage any further we can decrease the maximum size of the tranlogs. Looking at the file

/opt/gemstone/product/seaside/system.conf

there is a lineSTN_TRAN_LOG_SIZES = 1000, 1000;I just changed it to

STN_TRAN_LOG_SIZES = 100, 100;

which makes tranlog files of 100MB in size. The files are smaller and become obsolete much earlier. We can remove them with the script above on a regular base. The script detects if nothing is to do. I just start it soon enough before the backup process starts to run.

Pier Sprint Eindhoven

Pier is a completely object-oriented CMS toolbox (this blog runs on pier). A lot of pieces are there waiting to be discovered in order to build your own CMS. Discovering the internals of pier is not an easy task to do if you are not a experienced smalltalker. To change the ways of approaching pier and to ease the first user contact with it a sprint around pier has been organized. The sprint will take place at the weekend 2nd/3rd of july in Eindhoven. Details you can find here. If you like pier and want to know it better or help improving it than register yourself and attend the pier sprint.

Simple XML Testsuite

Building REST interfaces became a frequent task in my daily work. I really like to stay within my smalltalk image (the world is good in here) but crossing the image boundary and spread some bits over the internet has this adventurous touch :)

Leaving the image means leaving the language as well. In order to speak a format that can be considered common there are roughly two alternatives these days: Json and XML. Json is fine for very simple stuff. It is smaller and looks nicer. But these are all of the advantages. I’m not a big fan of XML but I get the job done with it. Implementing a remote interface involves two sides and most of the time that means two parties. To avoid problems in communication early I like to write XML schema files.

It is a proper definition of the negotiated format and every party is able to test a format without the involvement of the counter-part. It often solved troubles for me because I can just validate a snippet of communicated data to state clearly whose fault it was if it doesn’t work (often it is not that easy that’s true).

Developing an XML schema file can be a rather complex thing to do. In Smalltalk I’m used to create some test cases to ease my work. But in XML schema it is not that comfortable. I need to test that a given format can be parsed and that the unwanted cases fail. To help me with that task I created a simple script that tries to behave like a testsuite does. In order to be able to reuse it I put it on a git server. Enough reasons to announce it in my blog. Some other people might find it useful.

You can find it on github

XML Binding for Magritte

A very late blog post. It is a really long time since I started the XML Binding for magritte project. I always planned to do a short blog post to at least announce it somehow. In the meantime I used it quite a lot and it works for me very well. If you are used to mangle magritte descriptions you can unveal some of the secret powers. A lot of work on the code has been done by Jan van de Sandt.

Magritte is a framework that allows you to meta-describe your model. Meta describing contains extra information of all sorts like types, constraints and relationships. The relationships of objects can be mapped on a XML schema. The XML binding for magritte is not targetted towards serializing whole object graphs. For this I use Sixx. The XML Binding is useful where simple objects need a simple mapping onto XML in a cross-language and cross-platform manner (call it internet :) ). I started the project because I do a lot of REST APIs. In a REST way of doing things you wouldn’t nest objects over multiple levels. Only composite objects would be contained nested in their containing objects. Aggregated objects would be referenced by their own URI. how to use:

Let’s assume a test class MXSimpleObject (the class is contained in the tests of the package)

MXObject subclass: #MXSimpleObject 
   instanceVariableNames: 'title abstract date'  
   classVariableNames: ''   
   poolDictionaries: ''  
   category: 'Magritte-XMLBinding-Test'

We have three instance variables that should be mapped to XML.(btw. in this example the test class inherits from MXObject. This is not necessary to use the XML binding it just eases some things like this demo)

Every variable needs a getter and a setter.

title  
   ^ title

title: aString   
   title := aString

generate getters and setters for the other two variables as well. Now we first add magritte descriptions for the instance variables.

descriptionTitle 
   ^ MAStringDescription new     
      accessor: #title;    
      priority: 100;    
      label: ''Title'';    
      yourself

descriptionAbstract   
   ^ MAStringDescription new     
      accessor: #abstract;    
      priority: 150;    
      label: 'Abstract';    
      yourself

descriptionDate 
   ^ MADateDescription new    
      accessor: #date;     
      priority: 350;    
      label: 'Date';     
      yourself

Now we have described our object with magritte. In order to add the XML mapping to the object we need to alter the magritte descriptions. We decide that the title should be title and date should be an attribute and abstract should be an element. To tell the XML Binding we want an attribute add beXmlAttribute to the magritte description. The name of the xml entity (attribute name or element name) is taken from the accessor selector (accessor #date will lead to an entity name ‘date’). The names of the entities can be changed. To give the attribute a different name add xmlAttributeName: to the description. If we want it to be an element we just add beXmlElement to the description. To change the name of the element add xmlElementName: to the description. The resulting descriptions look like this

descriptionTitle 
   ^ MAStringDescription new     
      accessor: #title;    
      priority: 100;    
      label: ''Title'';    
      beXmlAttribute;      
      yourself

descriptionAbstract   
   ^ MAStringDescription new     
      accessor: #abstract;    
      priority: 150;    
      label: ''Abstract'';    
      beXmlElement;     
      yourself

descriptionDate 
   ^ MADateDescription new    
      accessor: #date;     
      priority: 350;    
      label: 'Date';     
      beXmlAttribute;      
      yourself

Finally we want the XML Element that is produced to be named “simpleobject”. For this we add a class side method

xmlElementName  
   ^ 'simpleobject'

We can now test if it works. Open a workspace and enter

MXSimpleObject description toXml: (   
   MXSimpleObject new      
      title: ''my demo binding'';      
      date: DateAndTime now;     
      abstract: 'this is just a demo about xml binding for magritte')

and you get

<simpleobject title="my demo binding" date="2011-05-25T18:38:05+02:00">  
   <abstract>this is just a demo about xml binding for magritte</abstract>
</simpleobject>

To be really useful we need to do a roundtrip test. Copy the produced string and recreate the object with

MXSimpleObject description fromXml: '<simpleobject title="my demo binding" date="2011-0525T18:38:05+02:00">      
<abstract>this is just a demo about xml binding for magritte</abstract> </simpleobject>'

and you should get an object back this is the same as the one we’ve started with.The XML Binding can deal with one-to-one and one-to-many relationships. There is enough testing code in the package that shows how it works. If you want to play with it you can get it from Lukas’ source site. Just load the

ConfigurationsOfMagritteAddOns2Gofer new   
   renggli: 'magritte2addons'; 
   package: 'ConfigurationOfMagritteAddOns2'; 
   load

followed by a

((Smalltalk at: #ConfigurationOfMagritteAddOns2)    
   project latestVersion) load: 'Magritte-XMLBinding'

Stone Creator Utility Released

In gemstone you can have a lot of different options if you set up multiple projects at your site. I tried putting a lot of software projects into the same image at first (is still good to start with). Then I used a multiuser setup that is the same image but complete separate object graphs by using different users for it.

But those have a lot of drawbacks regarding dependencies and manageability of the projects. So I ended up with a multiple stone setup. It is not really hard to set up another stone but gemstone does not include a utility to help you with. There are some written in ruby. It would have been a great reason to learn ruby but this time I tried not to postpone everything until I’ve learned enough ruby to be able to manage some scripts. Another reason is that if it works with bash I favour it because it is closer to the system and less bloated. So I did a bash script that creates a skeleton directory for a stone to run within. It sets all environment variables right and provides a system V start/stop script so you can have your stone started if the machine boots. You can it at github