Dear Jan:

Unless I don't understand what do you mean by more work in "activities", I think I agree with you in the first and third point. However, and specifically in terms of the programmatic data (outputs, results, etc.), I think there is not many visualizations created to show this (or at least I don't see them on Aid Info Labs http://www.aidinfolabs.org/tools/). 

Dear Cristian

thank you for responding, and apoligies for taking so much time to answer.

Well we agree on having data on point 1, the environment where activities take place (I might mention that geocoding would pe part of it) and on point 3, the output/outcome/impact. Naturally, work would have to be put into properly defining the details. We think  point 2 is crucial, if IATI data shall not remain of bureaucratic interest only: IATI has the potential of becoming a study and learning ground to define factors of success and failure. For this, it is necessary not only to know the funds spent, but to know the activities linked to them through a proper classification system. I have no space to explain further in this post, but some hints may be found in attached article.

I

Hi Jan

Could you give some specific examples of what you mean by "a more systematic and detailed 
classification system should be applied".

This is the right place to raise this issue, but for your suggestion to be taken forward we need something more concrete to work with.

IATI Activity Data

Dear colleagues,

as promised some time ago, I will try to explain the rationale behind my statement that activity data for given projects should be more explicit, and more systematically structured. The claims is quite relevant in our understanding, and I request that our reasoning is not dismissed before our paper has been carefully read. I will try to be as short as I can.

Is there a financial policy and statistics bias “in IATI”?

IATI is the only initiative at this time which tries to bring as many donors as possible and their projects or interventions under one standardized umbrella. The standardization brings together many interventions which otherwise would remain unrecorded and will allow much progress in coordination of many aspects. It might also, and this I would not tire to reiterate, be a unique opportunity to compare and evaluate donor’s priorities and resulting activities worldwide. And this in the field of individual and concrete interventions, which have never put to such a far reaching test and comparison as may become possible with the properly defined (and implemented) datasets in the IATI standard which this reasoning is about.

For the moment, the standard appears in large stretches formulated for an audience which is primarily interested in data processing and related technologies, and quick availability of financial figures. For field-minded people like us it is difficult to detect whether issues we are interested in are treated to an extent that we think is necessary.     
Maybe we are wrong , but, among other things, our group has the impression that the focus is primarily on administrative and financial planning matters and much less on matters of “how” an intervention has been set up, addresses problems, and interacts with the people concerned (the “stakeholders”, “clients” or “target groups”), among others. But exactly these are the issues that we consider crucial- yet they are so often missed or treated marginally in the development discourse that we fear that this might happen in the development of this promising standard once more.

Areas where more focus should be considered

We think that, if properly structured (and implemented) in the category “activity” and two or three categories related to these, the standard could achieve much more: It could become an invaluable learning resource on how to set up projects, and how not to- by opening access to the inner workings of huge numbers of success and failure stories. The categories are:

1. activities conducted during the intervention (including investment in hardware)
2. setting where the activities take place (that is: geographic location, and people concerned)
3. results obtained (and their relation to activities) in terms of output- outcome- impact.

To some lesser degree the objective(s) of an intervention may also play a role. But this is another story which may be left aside for the time being. All three of the mentioned points seem already to be part of the standard, but not to such a degree that it would be sufficient to be effectively analyzed and thus produce added value for analysis and learning.    
Naturally, fruitful analysis means also to relate “activity” with some other aspects, e.g. the funds disbursed, timing of funds, and others. Some are more or less covered. In order to not complicate our reasoning, we would like to leave them aside.

It’s the individual project that counts

The IATI standard addresses the level of the individual project. This is quite appropriate and must be put to maximum use.

If we want to relate activities to results, we need to focus on the individual intervention or project where a direct link between activities and results can be assumed. This linkage is a key element in data collection and analysis. In fact, I cannot emphasize enough how important this focus is.

The reason is simple: Quality of results, or, put simply, success or failure can only become a learning experience, if the element of success (or failure) can be as closely related as possible to specific causes. And while these causes may be multiple, it is the activities in the intervention aiming at results that we will want to focus on in order to learn what should be done in a given context, and what not. It is the water outlet built at the right time in the right place with the right technology which makes the difference, and not a large budget with which ten wells have been built at random of which one survives somehow without us understanding why it was this one and not the others.

An example: Three interventions compared

Let me demonstrate with a simple example what can be deducted by looking at “activities” comparing three individual interventions which occur in similar settings. All three may be characterized by the concerned funding agency in a typical way using “terminology” we have found frequently in such projects. Though simplified, the following examples draw from many real experiences we have seen:

Let us now look at what was behind our three sample projects:

Project 1:

In this project, two years were spent in order to plan, build and equip a school building along the latest guidelines of modern learning, and a diesel generator was installed to provide power for the school lighting, for airconditioning the director’s offices, and for modern teaching and office machinery. In the third year, staff was recruited and equipment was bought.                 
During the fourth and fifth years, 50 students per year were trained in the modern facility. The second group of these could not finish as by mid-last (5^th) year a row between director and staff had broken out as regarding allocation of funds to salaries, books, and the diesel generator. Teaching was halted during part of the time.           
At the end of the 5^th year, 30 of the students had found a job, the other 70 returned to their family’s farms.

The school applied for an extension of the project as running (and maintenance) costs were far above what could be earned from school fees and services.

Project 2:

In this example, an existing government building was rehabilitated and enlarged, and main funds were invested in having a renowned agency design, edit, translate and print a set of top-notch textbooks for a number of curricula expected to be of relevance. Also, 10 teachers were trained and given scholarships abroad in order to acquire the latest teaching skills.   
Teaching started in the 3^rd year to a tune of 100 students per year. At the end of the 5^th year, there were 250 graduates with diploma (of a total of 300 students, the rest being drop-outs). Of them all, 60 had found a job in the profession they had learned, and another 60 had travelled to the capital and found some job there.      
At the end of the period, farmers said that it had become slightly easier to sell their produce, and some refrigerators and TVs had been bought, while a mechanical repair shop and a few bicycle repair businesses had come up. The number of applicants, however, remained stagnant, and fees did not cover running costs.

A new phase was in planning, with reduced number of teaching staff, higher fees, and some of the curricula discontinued.

Project 3

In the third sample project, an existing building was renovated adding large workshop space for practical training. Low-cost materials and local labour were used extensively. In cooperation with local people, a baseline survey on local well-being was conducted. Main investment in the first 2 years went into thorough training of teaching staff, support to private businesses in order to receive trainees for on-the job training, and into set-up of a joint governing body where local stakeholders from administration and the wider public participated. Main task was the joint development of curricula on basis of local priorities.                                                 
Training started gradually from the second year, and at the end of the 5^th year, some 350 trainees had completed their courses. Of these, about 100 set up their own small business, while most of the others had found a job.                   
A field survey at the end of term allowed comparison with the baseline, and it showed better availability and significant reduction in cost for goods and services, and visible poverty reduction. 
The scheme was viewed a success, running costs were covered by fees and services delivered by the school’s own facilities.

A new phase was considered in order to spread the concept in other areas, adding a microcredit scheme aimed at graduates intending to set up new small scale businesses.

Need to trace analysis to the individual project and its specific activities

The three projects in the section above did obviously differ in their impact, although objective, budget and duration were the same. The differences might be summarized as follows (0-3 possible points):

Table 1: Simplified rating of the three interventions

Two things are demonstrated by table 1:

1. a given amount of finance can create completely different results even when invested in the same sector and with the same objective. This is exemplified by the difference in impact between the three projects.
2. differing results (success and failure) can only be reasonably understood if they can be traced back to the individual project and its activities.

Both support the key statement: The difference is not made by the money alone (whilst it remains a necessary fuel for action), but it is the way an intervention is designed and carried out. Development strategists and policymakers would have an interest in being able to draw upon such individual experiences.

Table 1 also helps to make the point why we need to correlate activities from individual projects, and not from groups of projects (all vocational projects of an agency, in a country, in a region) which cannot be individually distinguished. Why go to such lengths? The last line illustrates the answer: The sum of all totals of all projects in the area (in our sample it is 3, but it might be any number) would give a completely misleading generalized interpretation when evaluating them which could read something like the following                      
“Vocational training projects cause high cost of buildings and hardware, not too much success in number of graduates, and unsatisfactory impact.”                                                                     
A decision maker could conclude that this type of project is not cost effective and decide to cut funding. The point would have been completely missed, without this even being realized. IATI could help to do better.

An outline of how “activities” could be structured for fruitful analysis

Of course, it is possible to simply allow that activities in a project are explained through a text. As we understand the standard, this is what can already be expected.

Text explanations, however, can be anything between short and precise and large and complicated. An analyst who is interested to correlate results to specific activities will have much work in trying to distill the relevant facts, and even so he cannot be sure that the main information has not somehow gone missing.

This is why we propose to structure the “activity” category into a matrix of activities that allows to appreciate where the main focus in the project is/was. In addition, an indication on the portion of funding (e.g. by putting in the percentage of total funds disbursed for the specific activity) that was assigned to the activity could be useful.

Activity denominators could be selected from a list (proposed in the IATI standard) and subsumed under relevant categories such as the ones proposed in table 2. The activities for the same three sample projects could then be structured like in the following table (which is a simplified example that may need further development). The structure is not a rating this time, but rather an aid to assign activities to given (sample) categories. In the example of table 2, the first column relates to the “activity” either as such or the investment category (which we think would need to be placed here if there is not another category in the standard where this may find its place). The other columns reflect the amount of effort or funding which goes into the respective activity group:

Table 2: Activities selected from a list and subsumed in a possible matrix

Of course, the above table is a rough simplification. But it would help to see what activities brought success, and what was a waste of funds. By a finer adaptation of categories, an even better picture of what happens in each project may be obtained. It is, however, not practical to produce too many categories, as then the assignment of activities may become too cumbersome for those who have to complete the data. 

As shown in table 2, it may be noteworthy that under the header “activities” plain investments are included. We have not seen a category in the standard where these found a place, and Being  an important part of any intervention, they might be suitably placed here. Doing so may be natural if we take into account that most projects display a mix of investment and activities, and that often projects do nothing than invest in a piece of infrastructure and then leave.

The purpose of this paper was to make the point that it is advisable to structure the activity category along specific lines which allow more profound interpretation. I hope, this purpose has been met.

We would be glad if our points are given the attention they deserve. Please contact us if you wish to deepen this issue.

Jan Stiefel

IDEAS AidRating Switzerland

 16.10.14