G## -*- coding: utf-8 -*- Revolution blahg

FOSDEM 2012 review

Wednesday, February 15, 2012

I went to FOSDEM this year. Thanks SUSE for sponsoring my trip! Here is a short review for the projects that I found interesting at this year’s FOSDEM.

SATURDAY

The Aeolus Project

Francesco Vollero – Red Hat

This is a very interesting project if you can go past how meta it is. It wants to be an abstraction over all the existing private and public cloud solutions. The aim of the project is to be able to create and control a virtual system throughout its life cycle. It can be converted from one VM image format to another and be deployed/moved from one cloud provider to another. Groups of images can be setup and controlled together. The way resources are managed and billed would also be cloud-independent.

It relies heavily on the DeltaCloud project.

Open Clouds with DeltaCloud

Michal Fojtik – Red Hat

DeltaCloud aims to be a RESTful API that is able to abstract all of the other public or private cloud APIs, allowing for the development of cloud-independent software. The project says it wants to be truly independent (esp. from Red Hat). It was accepted as a top-level Apache project.

DMTF CIMI and Apache DeltaCloud

Marios Andreou – Red Hat

The CIMI API is a specification for interacting with various cloud-resources. A lot of very big companies are part of the DMTF Cloud Management Working Group: Red Hat, VMware Inc., Oracle, IBM, Microsoft Corporation, Huawei, Fujitsu, Dell. It is currently being implemented as part of the DeltaCloud API. The presenter also showed some implementation details: a lot of the code is shared between the DeltaCloud and the CIMI API.

Infrastructure as an opensource project

Ryan Lane – Wikimedia Foundation

The talk went into some detail about the whole Wikimedia setup. It is built on top of open source projects and aims to be entirely free and available to anyone who wants to know more about it. The speaker presented some of the issues that the Wikimedia organization faced when they decided to give full root access to their machines to volunteers and how to allow for different levels of trust.

Orchestration for the cloud – Juju

Dave Walker – Canonical

Juju is a system for building recipes of configurations and packages that can then be deployed on openstack/EC2 systems. The project aims to integrate with tools like chef and puppet to be able to manage deploying, connecting, configuring and running suites of applications in the cloud.

OpenStack developers meeting

This was a rather informal discussion. 4 major distros were present: Fedora, Ubuntu, SUSE and Debian, but also some other contributors. Upstream asked about the problems that distributions face, some minor one-time occurrences were discussed briefly. Stefano Maffulli, the openstack community manager was also present and there were some heated discussions about the way the project is governed. There are still a lot of things being discussed behind closed doors. Negotiations about the future of the project and fund-gathering is done with only a few big companies at a very high level. The community, on the other hand, was very vocal about wanting to rule itself with no enterprise interference.

Rethinking system and distro development

Lars Wirzenius

Advanced the idea of maintaining groups of packages, all locked at a specific version. Having the maintainers always know which combination of versions a bug comes from would make the whole environment easier to replicate and the bug easier to reproduce. This would also, supposedly, reduce some of the complexities of dealing with dependencies.

These groups of packages would be built directly from the upstream’s sources, following rules laid out in a git repository. The speaker also said he wants to get rid of binary packages completely.

If this were to be implemented, distributions could write functional tests against whole systems (continuously built images), rather than individual binary packages and ensure that a full configuration works.

Someone from the audience mentioned that a lot of the ideas in the talk are already implemented in NixOS(nixos.org) (which looks like a very interesting project in itself).

SUNDAY

Continuos Integration/ Continuos Delivery

Karanbir Singh – CentOS

The speaker discussed the system which CentOS uses for continuous integration. I liked their laissez-faire approach to which type of functional test language they should be using. They basically allow any type of language/environment to be used when running tests. The only requirement is that the test returns 0 on success and something else on failure. Anyone can write functional tests in any language they want (they just specify the packages as requirements for their test environment). People can choose to maintain different groups of packages along with the tests associated to them.

The Apache Cassandra Storage Engine

Sylvain Lebresne

A lot of interesting concepts about the optimizations that were made in the Cassandra project in order to speed up writes and make reads twice as fast (almost as fast as reads): different levels of caching, queuing writes, merge sorting the read cache with the physical data on reads etc.

Freedom, Out of the Box!

Bdale Garbee

An interesting project about making a truly free easily available software as well as hardware system. Some interesting concepts are used in this project like GPG keys for authentication, but also for the trust required to provide a truly decentralized peer based network, free from DNSes.

I’ve been to a few other talks that I can’t remember anything from either because of the bad quality of the presentation or because I didn’t have the prerequisite knowledge to understand what they were talking about. Next time I should also take notes.

A lot of the talks were recorded and are available over here (with more coming): FOSDEM 2012 videos. The quality of the recordings (esp. in the main room) is sometimes even better than being there live. The voice is clearer and there is no ambient noise. Also, as it was really cold in most of the rooms – I had to keep my jacket and hat on.

Tags: programming databases foss fedora
Comments post separator

SQL and Relation Theory Master Class

Monday, September 26, 2011

This video course is perhaps the best way to meet the famous C. J. Date and his astonishingly comprehensive style. The lectures are a great introduction to database theory while at the same time they lay a very solid foundation for any database practitioners or theorists. The author introduces some very useful theoretical notions that are essential to grasping the more subtle concepts of database design and he does so in a high-class fashion.

C. J. Date’s style of explaining and teaching, which can also be seen in his books, is didactic and very thorough while at the same time astonishingly clear. Many times while reading the book that these videos are based on and even afterward while watching the videos, I had to stop in order to reflect at the great volume of information that I had absorbed in a surprisingly simple manner. These videos are full of very deep notions about databases and can really benefit from reviewing at a later time, just to cement the knowledge or reflect on certain topics which come up during everyday practice.

C. J. Date sets out to demolish SQL as a language fit for relational theory and databases in general. While going through all the database theory concepts he presents the ideal case and an ideal query language (actually not ideal, but as he demonstrates, the correct ones) contrasting them to generic SQL. He also posits and sets out to prove, in a very interesting argument, that relational databases are the only way to store data and all other data models will not endure.

These are the days of NOSQL databases, but I think that the information contained in these lectures will be useful for a lot more time and in a lot more settings than just conventional SQL databases that are used in the majority of current systems. I oftentimes find myself thinking in relational terms even while designing the redis data model that I’m currently working on.

The only problem I have is that I sometimes felt that the lectures were a bit dull. It is also possible that I got this impression because I was watching too many without interruption :). While the content of the lectures is excellent, the presentation could be improved. Often times I felt that the audience present in the classroom could have done more to improve the dynamism of the lectures. It seemed that the only reason why they were there was so that the presenter wouldn’t feel alone. I would have enjoyed more challenging questions and especially some skeptical comments from industry veterans perhaps. I’m sure those would have led to very interesting debates considering the high class of the lecturer and presumably, the attendants.

Tags: foss programming databases books
Comments post separator

Copr final report

Tuesday, September 28, 2010

Fedora Summer Coding is now over for me and I’m really glad of what I learned and coded this summer.

Our initial goal was to develop a TurboGears2 Web app and JSON API for Fedora Copr. When finished, Copr should provide everyone with a place to build Fedora packages and host custom repositories for everyone to enjoy. This is a project that should prove quite popular in the Fedora Community when it gets released and I’m glad to have played a role in its development.

At first I worked on the web app, modeling the database and the relationship between coprs and repos and packages and then developing the JSON API. When the midterm came, my mentor and I decided that I should also contribute to the other parts of Copr. The original schedule had a simple command-line client planned, but we went further than that. In the end all of the functionality of the JSON API also got implemented in a client library (based on and very similar to python-fedora) and in a command-line client. I also got to dive into python-fedora’s and repoze.who’s internals in order to get basic HTTP authentication working for TurboGears2.

My latest work has been on the func module. This is the buildsystem part of Copr. Func minions running this module will be commanded by headhunter (Copr’s scheduler) to build packages in mock and then move them into repositories. The module also creates, updates and deletes package repositories and will check the built packages for Fedora conformance (e.g. licensing) – this last part is not yet implemented. I got to play with virtual machines and func and mock and createrepo.

There is a more synthethic overview of all the different things that got implemented on the wiki.

Overall, I’m really glad of what I learned this summer. This project really got me involved in a lot of different levels of the architecture of a web service and a lot of different technologies. Some of the things I worked on looked really scary at first, but as I went nearer and read more code the mist slowly vanished.

My mentor, Toshio Kuratomi was great as always. This isn’t the first project I’ve had him as my mentor. He was always there to talk to and always had great answers to all of my questions. He had great patience in answering and explaining anything I asked about. Our discussions were mostly about the architecture of the app we were building, but he also gave me great tips on the inner workings of python-fedora or on deploying the web app. I felt I had a lot of liberty to decide the way things will get implemented. Regardless of whether we will ever work together again, Toshio will always be a great inspiration for me as a programmer and as a person.

Tags: fedora copr fedora-summer-coding foss
Comments post separator

FSC: moving to the buildsystem

Monday, September 6, 2010

I started working on the buildsystem part of copr this week. Right now, I’m still getting familiar with func. That’s what we’ll be using to communicate with the builder machines: get them running errands and get back status reports at any time. I spent a lot of time getting a virtual machine setup with libvirt; networking especially was a pain (mostly because of my pppoe connection I think).

One nice feature of func that I think we’ll be using a lot is the async mode. A mock build takes a lot of time, what with all the yumming and compiling. So starting a task via one of the user interfaces and then choosing whether or not to keep watching it and what to watch for will probably be an essential part of the buildsystem’s functionality.

In the meantime, we’re slowly getting resources for the deployment of Copr. Toshio got a running instance of the current state of the TG app on publictest1. It looks just like a quickstarted TG app, because it doesn’t have any WebUI. But it can CRUD coprs, handle dependencies between them, handle permissions and CRD packages. Most of the functions require a FAS account, but you don’t need one to see a list of all the coprs, or a list of packages in a copr.

Tags: copr fedora-summer-coding foss
Comments post separator

the Copr client part II

Monday, August 30, 2010

I spent this week finishing up the copr client. It now supports all the functionality that the Copr TG API supports. It’s not much, but it’s a starting point.

I spent a lot of time trying to understand the way repoze.who works and the authentication plugins that we’re using for the python-fedora FAS authentication plugin. I finally understood it, I think… The Fedora client library didn’t support basic HTTP Authentication for TG2 apps so I had to figure out how to integrate that into our authentication plugin. It was quite fun all in all, repoze.who has a very interesting way of doing authentication and writing wsgi middleware is always exciting ;). This patch will hopefully go upstream to python-fedora now.

This next week I’ll probably start working on the buildsystem part of Copr. There are a lot of new things to learn there.

Tags: copr fedora-summer-coding foss
Comments post separator

the Copr client

Monday, August 23, 2010

This last week I started working on the command line client to Copr. Luckily, the python-fedora already has a lot of code in place to make the task of writing clients for TurboGears apps a lot easier. Some of the apps in infrastructure are already using this library, which make for some good examples.

So I’m building a client library and a client um… command line client. The command line client is basically one big argparse application that calls the functions in the client library and sometimes does a bit of formatting on the output. The client library implements a fedora.client.BaseClient that mostly just calls json methods on the Copr server.

It’s all pretty simple. The hard part is deciding what the command line client’s interface will look like. In argparse parlance, which ones should be the positional arguments and which should be the optional arguments. So far I’ve been inclined to use something that looks like a VCS’s interface. Here’s what it looks like so far:

$ python client/bin.py -h
usage: bin.py [-h] [-v] [-u USERNAME] [-p PASSWORD] [--url URL]
              {info,edit,create,list,delete}
Command line tool for interacting with Fedora Copr
positional arguments:
  {info,edit,create,list,delete}
    list                list all the available Coprs
    info                get information about a specific Copr
    create              create a new Copr
    edit                edit an existing copr
    delete              delete an existing copr
optional arguments:
  -h, --help            show this help message and exit
  -v, --version
  --url URL             provide an alternate url for the Copr service
authentication:
  -u USERNAME, --username USERNAME
  -p PASSWORD, --password PASSWORD

Right now, all the copr functions are top-level. I wonder if I’ll have to create a deeper level of nesting when I start implementing package-related functions.

I’m also having a few problems with the BaseClient that I’ll probably have to solve this week. All of the other client libraries were written for TurboGears 1.x and it seems that authentication has changed in TurboGears 2. There’s also no support for HTTP PUT and DELETE which I would like to use since I used RestControllers in the API. I also had to write a patch for file upload support; that seems to work well so far.

Tags: copr foss fedora-summer-coding
Comments post separator

Ruby koans

Sunday, August 22, 2010

I had a great time today with Ruby Koans. It took me about 5 hours in all. A good way to spend a Sunday afternoon I suppose.

These Ruby Koans are a great way to go on a quick journey through a lot of Ruby’s common features. You basically have to edit tests in order to get them working. It’s mostly reading tests actually, but the fact that you have to fill in some blanks keeps the mind from wandering. There are also a couple of exercises which imply a bit more coding.

I have a good knowledge of Python and have worked with Ruby in the past on a little Rails project. I had forgotten anything I knew about Ruby though. Yesterday, I don’t think I would’ve been able to write a foobaz in Ruby without looking for help online. This proved to be a welcome refresher. Solving these koans gives a great tour of Ruby. As I went through them I kept thinking of how I would do those things with Python. I really like Python’s philosophy and maybe solving all these ruby koans has made me appreciate Python’s simplicity and predictability a bit more. Ruby allows for a lot more flexibility however and the koans left me to wonder at what amazing feats this language could accomplish.

I wouldn’t recommend this to a beginner however. While I think I now have a pretty good idea of what the language can do, there were no whys or recommendations about all these features. Maybe it would be a good starting point (or a dive) for someone coming from a similar language (like Python), before moving on to a good Ruby book. The website claims that they teach culture in addition to Ruby. I would’ve liked more of that. Maybe it was too subtle for me, but I didn’t notice anything other than some references to oriental philosophy: test_assert_truth has damaged your karma. You have not yet reached enlightenment ...

There are a lot of ports of the Ruby Koans. There’s one for python and there are also a bunch for functional languages: Clojure, F#, Haskell and Scala. These look like a lot of fun, maybe I’ll try them next week.

Tags: foss hacking
Comments post separator
Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License.