LATEST articles

Generating fixtures with Alice

A common problem in software development, is that you need data to work with. This is especially true with data-oriented websites. Working with an empty database leads to all sorts of unexpected problems cropping up once the site goes live and receives real world data. Performance issues, visual bugs due to missing fields, too long or too small texts, etc.

The best way to address this is to have fixtures, fake data you use while developing or for automated testing of the website. However writing these fixtures can be a cumbersome process which leads to people postponing it and sometimes outright skip this step.

It used to be that Doctrine1 could load fixtures from yaml files, and symfony1 also had facilities there, but none of this has been made available to Doctrine2 or Symfony2 yet, or at least not in a very usable form.

Instead of writing a plugin for either of those, I set out to write a generic library to easily create objects from yaml files. It can be used with any framework or ORM, and also integrates the great Faker library to generate fake/random data. The library takes a few opinionated turns, so it might not be for everyone. However I am sure it will save you a great deal of time should you decide to work with it.

The library is called Alice, after the common Alice and Bob placeholder names, and you can of course find it on GitHub, including extensive documentation.

Let me know what you think of it!

October 29, 2012 by Jordi Boggiano in Development, News // Tags: , , 8 Comments

A python solution to a secure backup of CouchDB via replication

This guest post was written by Reto. He works at Bluevalor, Nelmio’s very first client ever. Reto is a crack at analyzing economic data using MATLAB and Python. But he can not help himself and sometimes enjoys working on our infrastructure as well.

When we started our project with Nelmio, Pierre proposed to use CouchDB as a container for highly dimensional data items we receive from our third parties. We have been happy with CouchDB so far. One of the nice features of couch is its reliance on sequence IDs to assure very easy synchronisation between different CouchDB instances. It is even possible to use these sequence IDs to set up synchronisation between say SQL and CouchDB, since there is a nice API to query for changes in the CouchDB server.

A very convenient way to set up a backup of the data is to just configure a second CouchDB on another machine and replicate the data onto that machine. There is a feature called “continuous replication”. This seems to imply that you would have to set up the replication only once… However there is quite a big drawback as of CouchDB 1.2.: If the server is restarted, the replications will not be re-initiated. Even worse, sometimes replications just break down without any apparent reason.

Update: If you set up the replication via the _replicator database it fixes the restart issue.

In short: CouchDB’s “continuous replication” is not reliable enough as a backup system.

I’ve written a small Python script that you can run as a cronjob to check if a replication exists for a list of CouchDBs. As a little bonus, I added email notification in case something is wrong, so you can sleep well knowing your CouchDB backup is still working. With this script, it should be viable to backup your CouchDB databases via replication. I’ve attached the code after my little fanboy praise of Python.

I’ve studied finance and basically taught myself programming for scientific purposes. I’m trying really hard to write good code, but sometimes, I lack experience because I do not have a true programming background. If you’d like to point out things i could do better in terms of form, structure or function, please comment!

I’ve worked extensively with MATLAB so far. However, recently I stumbled over Python as a language for scientific computing and I’m absolutely loving it, so I would like to take the opportunity to praise on Python a little:

There are various reasons to use Python for scientific computing:

  1. high level language (good productivity, easy to learn for people like me)
  2. general purpose and object oriented (can interface with everything, bigger projects possible)
  3. beautiful, easy to read syntax
  4. ability to interface with low level languages if speed is first priority
  5. very rich libraries that support scientific computing needs
  6. open source (the MATLAB commercial license is 10-20k CHF depending on toolboxes)

With the open source packages numpy, scipy, ipython and pandas, Python pretty much trumps over every other scientific toolbox (R, MATLAB, Mathematica) while remaining super easy to use.

Especially pandas (an open source library that was developed at a hedge fund – true story!) improves data handling of time series ten-fold. I truly believe that if you need to do research with time series data, Python with pandas is the future.

So if you ever run into a problem where you need to do a lot of data cleaning and wrangling, look at pandas. There is a very good book called Python for Data Analysis written by Wes McKinney, the main developer of pandas (albeit only released as an “early release”).

Now to the code: Note that you will need the couchdb library to make this code work, so either install couchdb-python in your Python folder, or simply put it into the folder of the script. Note that I’ve only tested it with Python 2.7. You need to configure the “CONFIG” part of the script, and you should be all set.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
import couchdb
import datetime
import smtplib
from email.MIMEMultipart import MIMEMultipart
from email.MIMEText import MIMEText

#-----CONFIG------------------------------------------------------
#source & target adresses
SOURCE = 'http://admin:pwd@host:5984'
TARGET = 'http://admin:pwd@host:5984'
#list of dbs (must have equal length)
SOURCE_DBS = ['db1', 'db2']
TARGET_DBS = SOURCE_DBS
#email credentials
GMAIL_USER = "gmail_user"
GMAIL_PWD = "gmail_pwd"
TO = "your email"
# set to False if no email desired
SEND_MAIL = True
#-----------------------------------------------------------------


class CheckReplicator(object):
    """
    Checks if a replication for a list of dbs on the target exists between two
    CouchDB instances.

    Input a connection string for the source and the target of the
    replication and provide two lists with the names of the databases you want
    to have replicated. (source_dbs[0] will be replicated to target_dbs[0] etc)

    Note that this class prints to console, so if you want to log progress,
    print output to file in cronjob.

    Note that all the dbs need to be created. It is smart to initiate the
    first continuous sync via futon or http api!
    """
    def __init__(self, source, target, source_dbs, target_dbs):
        db_equality = len(source_dbs) == len(target_dbs)
        assert db_equality, "source length must equal target length"

        self.source = couchdb.client.Server(source)
        self.target = couchdb.client.Server(target)
        self.source_string = source
        self.target_string = target

        self.desired_reps = zip(source_dbs, target_dbs)
        self._check_connections()
        self.active_reps = self._get_active_reps_on_target()

    def check(self):

        if self._check_if_all_desired_reps_exist():
            print str(datetime.datetime.now())[:-7] + " ok"
        else:
            self._fix_replications()

    def _check_if_all_desired_reps_exist(self):
        res = True
        for d in self.desired_reps:
            if d not in self.active_reps:
                res = False
        return res

    def _fix_replications(self):
        for d in self.desired_reps:
            if d not in self.active_reps:
                source_str = self._build_source_string(d[0])
                self.target.replicate(source_str, d[1], continuous=True)

        self.active_reps = self._get_active_reps_on_target()
        if not self._check_if_all_desired_reps_exist():
            raise EmailError("""
                             could not replicate all targets. Please
                             check if the couch instances are running
                             and all the dbs are created!
                             """, SEND_MAIL)
        else:
            print str(datetime.datetime.now())[:-7] + " replicators created"

    def _build_source_string(self, db):
        string = self.source_string
        if string[-1] == '/':
            string = string + db
        else:
            string = string + '/' + db

        return string

    def _get_active_reps_on_target(self):

        tasks = self.target.tasks()

        #parse source and target of the task string
        #from the replication information
        active_reps = list()
        replications = [t['task'] for t in tasks if t['type'] == 'Replication']
        for r in replications:
            first_split = r.split('/ -> ')
            target = first_split[-1]
            second_split = first_split[-2].split('/')
            source = second_split[-1]
            active_reps.append((source, target))
        return active_reps

    def _check_connections(self):
        try:
            self.source.version()
        except:
            raise EmailError('could not connect to source', SEND_MAIL)

        try:
            self.target.version()
        except:
            raise EmailError('could not connect to target', SEND_MAIL)


class EmailError(Exception):

    def __init__(self, value, send_mail=False):
        self.value = value
        if send_mail:
            self._mail('Watchman Error', value)

    def __str__(self):
        return repr(self.value)

    def _mail(self, subject, text):
        msg = MIMEMultipart()

        msg['From'] = GMAIL_USER
        msg['To'] = TO
        msg['Subject'] = subject

        msg.attach(MIMEText(text))

        mailServer = smtplib.SMTP("smtp.gmail.com", 587)
        mailServer.ehlo()
        mailServer.starttls()
        mailServer.ehlo()
        mailServer.login(GMAIL_USER, GMAIL_PWD)
        mailServer.sendmail(GMAIL_USER, TO, msg.as_string())
        # Should be mailServer.quit(), but that crashes...
        mailServer.close()

#run it!
if __name__ == '__main__':
    try:
        check_replicator = CheckReplicator(SOURCE, TARGET, SOURCE_DBS, TARGET_DBS)
        check_replicator.check()
    except:
        raise EmailError('program code failed', SEND_MAIL)
October 1, 2012 by Nelmio in Development // Tags: , 3 Comments

One Year of Nelmio

A bit over a year ago Jordi and I took the plunge and gave birth to Nelmio. In this blog post I will tell the story of this last year, or at least try. It feels like this all started yesterday but many things happened.

Baby steps

We both quit our job at Liip in the end of April 2011. At that moment we were pretty much clueless on how to proceed. We had the necessary CHF 50k to incorporate here in Switzerland. I moved to Zürich and started to learn about all the legal steps we’d have to go through. May 2011 was over before we even realized it. Jordi had gone to holidays, we attended various conferences and managed to incorporate the company. Nelmio AG was born on June 16th.

Next I started to work on a tool to convert Markdown into beautiful PDFs using FOP. I had done a lot of XSLT before and had a lof of fun with that. We use this tool to write our offers, contracts, letters and invoice. During this time we moved into the Citizen [co-working] Space and the Summer was spent writing many offers for smaller and bigger projects.

Even though we had no contract signed, we hired Igor on August 1st 2011. Having an employee on the payroll and 2 kids and a wife to support at home was really scary at that time. I constantly questioned the feasibility of having our own company, of being able to survive. Luckily, the day our bank account got empty, we signed our first contract. This happened on August 18th 2011, exactly 3 month after incorporating. The client paid 50% of the project costs up front and we were able to pay our salaries with earned money for the first time ever!

From them on things started to move a bit faster. Potential clients got home from their holidays and started to return our emails. We got 2 more contracts signed and worked a lot to fulfill all the things we promised.

We love our clients

I think we really got lucky with our clients. I do not really know how this happened, but we managed to work with clients that are amazing from day one on. They trust and respect us. We teach each other a lot. They are willing to take risks, work with cutting edge technologies and go places no one has ever gone before!

Currently we work on a “Web Of Things” project for Simple Complex. We receive data from various types of sensors and have to compare them, represent the data in a meaningful way, set up alert systems. I can not really tell you more about this project quite yet, but it is a lot of fun to work on. We render beautiful charts with Raphaël, try to figure out if the sensors are lying to us, and build a client side application using Backbone.js that talks to a JSON API written in Symfony2. Actually, we use Backbone.js on almost all our client-facing applications and write all our APIs with Symfony2.

Then there is Bluevalor, our first client. Initially, we were supposed to only do the frontend work on that project. Unfortunately our client got really unlucky with their Java backend partner. The backend was supposed to aggregate massive amounts of data in order to process it with MATLAB but a functioning prototype was never delivered. We stepped up to write a proof of concept and proved that we could do the backend job as well for a fraction of the price. It is a challenging project in which we process several hundreds of megabytes of data per day. We use cutting edge technologies like the Go language, CouchDB and Buster.JS.

Another long-term client of ours is Cinergy. They own multiple Swiss platforms like cineman.ch, teleboy.ch and moonwalk.ch. In the year we worked with them, we integrated their existing team and worked together on pushing technical boundaries to bring their platform even further. We mostly worked on Teleboy, their TV application for the web and mobile devices.

Among others, we have worked on two frontend-only projects, helping local.ch and Squirro when bootstrapping their Backbone.js apps. Both these applications are still under development in-house after we left their teams. We are really looking forward to see these apps on the market soon!

Of course we could not have done it without our clients. We have had a lot of fun working with them and are really glad we had the chance to meet them!

Going remote

After one year, the company takes a massive pivot. William, our intern, will be leaving us to pursue his studies in France and Igor has left to discover other new things. We really enjoyed working with people an did not want to go back to being just the two of us.

We started looking of people but unfortunately it is really hard to find developers here in Switzerland. There is too much work and not enough people with skill. We started to consider remote work. We set up a job advertisement on our blog and had many applications from all over the planet. We got really lucky for find three people who are smart and willing to join Nelmio:

Daniel and Francesco are both from the northern part of Italy and will join us on September 1st. Samuel, who lives in Lisbon, has already been with us for 2 weeks now.

We will all be working from our cities and have daily meetings in Skype. Every couple months, we will travel to a European city and work together for a week. The two first trips are planned already: Barcelona and Berlin.

Things change all the time and having a company is a really crazy thing. But until now we have been pretty lucky and even though I spend more time talking to clients, doing bookkeeping and writing contracts than coding, I have never regretted this move.

Let’s see where this adventure is going to take us over the next 12 month!

July 13, 2012 by Pierre Spring in News // Tags: 8 Comments

A Tour of Go

Last week we had to write what a small specialized HTTP client that connects to a server and issues GET requests to fetch data. Simple enough.

The only issue is that the data provider required us to use maximum one connection at any time, using HTTP pipelining to issue all the GET requests in the same connection and then block until they respond with new data. In practice this is not very complex, but PHP (our go-to hammer) is not really shining in this area.

All the HTTP abstractions around curl/http streams like Buzz or Guzzle do not seem to support pipelining (if I missed that, let me know), and doing it by hand is not particularly fun. The http extension does support that through some undocumented feature though, but I didn’t have that extension at hand, so I skipped that. In any case, PHP is not really the best tool for doing this sort of long running script, although it has gotten much better in recent years. So we looked for something else, and long story short, we decided to try Go. It is a new-ish language that is aiming to replace C/C++/Java for systems programming, mixing and matching features from everywhere while focusing on language terseness.

So what about Go?

Well it is interesting. I just opened an editor and within a few hours I had a first prototype running, a few more hours of polishing and tweaking later I had a finished product that has now been running for a week. It handled a few tens of thousands of entries, and especially did not crash once ;) The standard library is quite complete for what I wanted to do here, and there is already quite a large ecosystem of extra libs available for more specific tasks.

The documentation. It is not awful but it is a young language, and it could benefit from having more examples and cross-references. On the other hand I started coding without reading any of the basics, so I had to learn a few things the hard way. One big plus though is that the documentation links to the sources, so you can very easily check out what something does behind the scenes.

Static typing is a big change coming from PHP which abstracts away a lot of this. However thanks to their light approach to typing, you don’t feel it too much. Basically using := for assignment you can define a variable and have its type automatically inferred by the value you assign it. That gives you typing without the verbosity. Then of course you get compile-time errors based on all this which is great for debugging. Overall this saved me from doing a bunch of mistakes, and did not cause so much pain.

The recover function is also quite a nice fit for the use case at hand. It allows you to recover from any fatal error, and then keep running some code. Using this I made sure the program never dies, because if anything bad happens it will error out and restart doing its job instantly. It reacts much faster than monit or similar watchdog programs would typically do since it is triggered by the application failure.

Finally I want to quickly touch on goroutines. It allows you to do parallel processing very very easily. All you have to do is prefix a function call with the go keyword, and it turns into a background task of sorts. You can then use channels to communicate and block between multiple goroutines. It is a great model and should allow people to write parallelized programs without the pain usually associated.

To sum up, Go is quite fun. I would not use it to build a large web app though. I don’t think the tools are there yet and we have a great ecosystem in PHP for that. But for simple utils I would definitely recommend it to anyone that feels like trying something new.

June 22, 2012 by Jordi Boggiano in Development // Tags: , , 3 Comments

An Appeal to All Package Managers

I work on Composer – a new PHP dependency manager – and it is now working quite well for managing PHP packages. We have a decent amount (and growing fast) of packages on Packagist. All is well. Yet, most PHP projects are websites, and they need some frontend libraries, be it JavaScript or CSS – I will use jQuery as an example that everyone can grasp easily. In some frameworks, you have plugins that bundle a copy of jQuery. Some people have also used Composer to hack together jQuery packages so that they can download it, and then they have some scripts to copy or symlink the files in the proper location. That is all very flaky, you end up with multiple copies of jQuery and if you are lucky you even get various different versions.

I have been thinking about it for a few months, and it seems like nothing exists out there. Every language out there has its own package manager, but everyone seems to be stuck with the same problem of frontend code. Obviously jQuery is used by virtually everyone. They can’t support Composer, Bundler, npm, pip and god knows what. Building a package manager for JS/CSS could work, but the community is huge and scattered and getting broad adoption is going to be very difficult.

The plan

As far as I can see, and given the way Composer works, it would be fairly easy to build a central package repository – similar to Packagist – where people can submit package definitions for frontend libs. For instance I could go and add a twitter bootstrap package, define the URL where you can download it including some placeholder for the version. Then all we need is a list of version that’s maintained. We could start by doing that by hand, or we can just point it to the git repo and it reads tags. That’s how Packagist works – except that it reads the composer.json file in the git repo to get all the metadata that in this case would be entered manually.

If we do this, we then end up with a central repository of CSS and JS packages, and we can integrate it in Composer, so that Composer packages can depend on jQuery and it just works. That would be a good start, but the great thing would be to get everyone on board. And I don’t mean everyone writing PHP. I mean everyone. The Ruby folks, the Python ones, Java, .NET, you name it. You all have package managers. All we have to do is agree on the API of the central package repository and on what metadata is needed. Then you can just add support for it in your package manager of choice, and we all benefit from the manual work put in to list packages. If it works, I’m sure some of the frontend packages will then add the metadata directly in their git/svn/.. repos so that we save on manual work. This would be a huge thing for everyone.

There are of course a few more details to settle regarding security and trust as well as exact package metadata, but I wanted to gauge the interest first, and then discuss further. I opened a frontend-packaging google group for that purpose, so if you are interested please join in. All it takes is a few open minded people and we could start one of the largest cross-language collaboration project ever. Sounds like fun!

May 31, 2012 by Jordi Boggiano in Development // Tags: , , , , 8 Comments

Nelmio is hiring

You might have heard of us. If not, here is
who we are:

We are a little company in Zürich, Switzerland. Our clients trust us. Our projects are technically challenging and mostly use cutting edge technologies such as Backbone.js, Symfony2, Redis and CouchDB. By keeping the company small we have the ability to carefully choose the projects we want to work on. This also gives us time to attend many conferences, both as speakers and attendees. And starting this autumn, we will be working on our product as well!

Right now, we would like to grow by one person.

Problems we currently solve are:

  • Finding best practices for pure JS applications using Backbone.js.
  • Writing REST APIs in Symfony2
  • Optimizing CouchDB and MySQL to cope with large amounts of data.
  • Web of things – Connecting physical sensors with the web. This is particularly fun and hard.
  • DevOps – Automating deployments and server setup.

Things we do:

  • Code. Code. Code.
  • Learn from clients.
  • Teach and empower our clients.
  • Learn from each other. A lot!
  • Care about the little things that make a difference.
  • Fun things, because.

Things we don’t know:

  • Designing beautiful sites. But we know people who do this very well.
  • Ruby, Python, Dart, … We write mostly JS and PHP, and contribute a lot to make sure these stay in the game.
  • CMS. We only work on tailored solutions.

So, what do you think? Would you like to join our team?

  • We do work mostly in English even if there is always someone who throws in some French or German words.
  • You should have good communication skills, and be able to act autonomously. The ability to learn and think is more important than your existing knowledge.
  • Remote work could be an option. You would need to be in roughly the same time zone and can come work in Zürich for a few weeks in the beginning. That way we could get to know each other.

Send us your CV, GitHub account or anything you think is relevant to hello@nelm.io.

May 30, 2012 by Pierre Spring in News // Tags: , , , 1 Comment

Nelmio is coming to a conference near you

Here is a quick update on conferences we will attend and speak at in the next couple months. If you are attending any, feel free to come and say hi!

Next week our entire team will be at jsDay and phpDay in Verona, Italy. If you haven’t got tickets yet, hurry up because it is about to be sold out.

Pierre will talk about Backbone.js, Igor has two talks about realtime apps with websockets and the Silex microframework and I myself will be talking about managing your dependencies with Composer.

In June there are two notable events. The first is SwissJeese. A local JavaScript-oriented conference co-organized by Pierre that will be in Bern. It is free and on a Saturday so there is really no excuse not to come if you have an interest in JavaScript. Register for free on eventbrite!

The second a week later is Symfony Live in Paris which we are always looking forward to.

If you get a chance to go to both phpDay and Symfony Live, you can follow Igor’s second talk about Silex, which will be more advanced given the audience is already more familiar with the Symfony Components. If you miss Verona you can catch up on my Composer talk in Paris as well. William will hold an introduction to the Propel2 ORM and even you have no interest in PHP I will present the Redis key-value store so you can come anyway ;)

We sure hope to see some old friends and make new ones there, so if you were hesitant to come, just register before it’s too late!

May 11, 2012 by Jordi Boggiano in News // Tags: , , , 1 Comment

An Update On Composer

This weekend we have been busy hacking on Composer in our office together with Nils Adermann and Volker Dusch. We wanted to push the project forward a bit faster than the odd free evenings usually allow, and I would now like to introduce the changes we made.

Development versions handling

The former master-dev and similar *-dev versions we used to have were causing quite a few issues, so we decided to overhaul that behavior in a way that allowed us to get more consistency and fix a few long standing issues. For example dev versions can now be locked to exact commit revisions, and they will update to the latest revision when you do an update, no need to delete them from disk beforehand.

Basically dev releases are now simply branch names with a dev suffix – for numeric branches which are comparable – or a dev prefix for textual names that are not comparable, like feature branches and master. There is no way to specify the version manually anymore in your repository’s composer.json, since that was causing potentially dangerous issues with feature branches conflicting with the original ones.

If your package depended on a master-dev version, you should now depend on dev-master. If your package depended on something like the Symfony2 2.1.0-dev version, this one also is now dev-master since it is in the master branch. Older feature branches like 2.0-dev which is the 2.0 branch and not master are unaffected by this change.

This change will break many packages out there that rely on -dev packages of any kind, and we hope everyone will update their composer.json files as swiftly as possible to make the transition less painful.

The Packagist version database had to be reset for this change, so things will look at bit empty for a couple of hours while everything is re-crawled. None of the packages are lost and you should not have to do anything except having a bit of patience.

Dependency solver stability

Nils and Volker have been doing big progress on bugfixing and testing the solver. Those are mostly highly technical details that I will not dive into here. But long story short many old bugs should be fixed, and then some. It may obviously have introduced regressions, so if you encounter any issues please report them with your composer.json file so we can easily reproduce.

Documentation

Igor has spent quite a bit of time on documentation, which you can see on github for now, and which should be migrated to getcomposer.org soon.

Packagist / GitHub integration

Another great new feature coming from a pull request by Beau Simensen is the ability to let GitHub tell Packagist when you push new code to your repository. This should make package updates almost instant. It should be integrated into the GitHub Service Hooks soon enough, so if you don’t want to set it up by hand you can wait a bit, otherwise you can grab your API hook URL on your Packagist profile page, and add it in your repository.

Repositories configuration

It seemed that the way custom repositories are configured was confusing, so we took the chance to make it a bit clearer. Basically names are dropped and it’s all stored in a flatter structure that’s easier to remember. Documentation has been updated on Packagist.

All in all it has been quite a productive week-end and we will continue working on a few things today.

February 20, 2012 by Jordi Boggiano in Development, News // Tags: , 6 Comments

jsDay – A Family Reunion You Always Wanted To Attend

Last jsDay in Verona was a blast. Seriously. To Jordi and myself this was one of the most memorable conferences ever.

And I am not talking about the sessions. Of course the talks were good and the jsDay team managed to attract some of the finest speakers to talk about everything JavaScript. But so do other conferences. The wonderful thing about jsDay is the community you become part a of as soon as you enter Hotel San Marco. Speakers and attendees alike share this hotel for 3 nights and 2 days and you feel you are part of the family reunion you never dared dreaming of.

Like-minded people sharing their passion. Cocktails at the hotel pool. A welcome quiet time in the Hotel Spa. Late supper in some of Italy’s finest Restaurants. A good laugh at Powerpoint Karaoke. And an incredible sense of belonging to this family that greets you with open arms and open hearts.

At Nelmio, we are now looking forward to the next jsDay in Verona and we just felt obligated to tell you about this little gathering that feels a bit like ours and that you too could be part of.

I sure hope to see you there on May 16th-17th 2012.

January 31, 2012 by Pierre Spring in News // Tags: 2 Comments

Composer: Part 2 – Impact

In the first part of this post I introduced Composer & Packagist. If you are not familiar with them please read part 1 first.

Impact

In this second part I would like to talk about a few things Composer could do for you, and the PHP community at large, once it is broadly adopted.

Common APIs and Shared Interfaces

You may have noticed that quite a lot of people are talking of and asking for more interoperability and cooperation between frameworks. It seems some PHP developers finally got tired of reinventing the wheel. That is great news. One way to provide this interoperability is through shared interfaces. The two main candidates there in my opinion are logging and caching. Two boring things that should just work, and where you always need tons of flexibility and tons of different backends, drivers, or whatever you want to call those. Almost every major framework and CMS out there have their own implementations of that stuff, yet none of them support all the options since there are too many.

The PHP Standards Group, an open mailing list discussing these interoperability questions has seen a recent proposal for a Cache Interface. One question raised was: How can those interfaces be distributed in each project that uses or implements them?

This is where I see Composer helping. Composer supports advanced relationships between packages, so to solve this issue you would need three parts (read carefully):

  • The psr/cache-interface package contains the interfaces, and requires a psr/cache virtual package.
  • Implementors of the interfaces (many libraries) all require the psr/cache-interface and also provide the psr/cache virtual package.
  • A framework that needs a cache library requires psr/cache-interface and hints the interface in its method signatures.

Then the user of that framework comes in, decides that he wants to use the Doctrine\Common cache implementation for example. By requireing doctrine/common, the psr/cache requirement of the psr/cache-interface would be satisfied. Both doctrine and the framework would use the interfaces from the psr/cache-interface package. No code duplication all over the place and everyone is happier. All those require and provide have version constraints on them, so the interfaces can easily be versioned so that Composer will not let you install things that do not work together.

Plugin Installs for Frameworks and Applications

Composer is built to be embedded in other frameworks, CMSs or other applications. Some parts are still a bit rough for that use case, but it is something that will be supported and encouraged. Reinventing the package management wheel is another thing that really should stop. Who am I to say this you ask? It is true, we are building a shiny new wheel as well. Yet I take comfort in the fact that we are trying to build a generic solution which will work for everybody.

Packages are easy to build – for those who insist on not reading the first part of this post: you drop a simple composer.json file and add the VCS repository to packagist.org. The goal is that building packages should be accessible. I would love it if TYPO3, Drupal or WordPress to name a few would use Composer as a library internally to handle their dependencies. The list of required packages does not have to be in a composer.json file, it can sit in a database just fine. That would mean that suddenly the WordPress plugin you are developing could depend on an external library to do some work, and you don’t have to embed the whole library code in your plugin’s repository. Autoloading would make it work magically as long as everyone respects PSR-0. Which brings me to my next point.

Promoting Standards

A few months back I was on IRC and someone linked his new library, who or what it was does not matter. I just noticed he used a home-made autoloader and asked him why he was not following the PSR-0 standard. The answer was “I just use a smarter autoloader, with fallback feature“. Now that’s great, maybe his solution is smarter in the way that it allows files and classes to be anywhere. But it messes with everybody else. No one can use that library unless they declare another autoloader just for it. Autoloading should really be a commodity that you do not have to lose time fixing.

By adopting and promoting the standard, I hope Composer will help raise awareness about it. If you follow PSR-0, Composer autoloads your packages. If you don’t, you are on your own. The more users start to rely on this, the more they will get annoyed when a package requires manual configuration to be autoloaded, which will put some pressure on the PSR-0 offenders.

Promoting Code Re-use

It is probably obvious, but having easy to use package management means you will use it more, and the more it is used, the more people will re-use and share code. I really hope to see many libraries pop up out there instead of the massive frameworks we had until recently.

This shift is already happening, the larger frameworks like Symfony2 and Zend Framework 2 have decoupled their internal components and it is now possible to use pieces of them individually. They start to look more like the PEAR repository, which is an aggregate of libraries that work well together, some depending on each other, but not all.

Single libraries out there are great but I see some value in these larger organizations enforcing some quality guidelines on their own code-base. In a way they act like brands. You know that if you use one of their packages you can expect a certain quality.

Renewed Interest in PHP

Overall, I believe that libraries like Buzz, Imagine and others can create a sort of DSL on top of the (sometimes really bad) PHP APIs. Many people have criticized PHP as a language for its inconsistencies and awkwardnesses. Fine. I am not going to argue with that. But I hope many of those people, if they are being honest, will agree that PHP as a platform is great. It runs everywhere, it does not require much configuration, it has an immense developer base.

If we have enough libraries that abstract away some of the language issues, I strongly believe PHP as a platform will have a bright future.

December 20, 2011 by Jordi Boggiano in Development // Tags: , 18 Comments