This item has been moved to the '2.02 Decimal Upgrade Proposals' forum for inclusion the upcoming decimal upgrade.

I have debated this at length with Herb Caudil at http://discuss.iatistandard.org/t/activity-results-and-objectives/172. 

In short, I believe the proposed modification would be beneficial but only in a slightly modified form. Firstly, reference to a standard indicator should be optional not required. This is because in many cases, good indicators must be tailored to their specific project. Secondly, the range of allowed choices of indicator libraries should be limited and needs to be decided carefully through a collective process. This is partly because use of certain types of library (such as unilateral donor-defined libraries) would risk running against the Paris Principles of country ownership, alignment and harmonization. It is also partly because measurement of results is problematic in detail, and aggregation of results can easily be misleading. Any scheme designed to enable aggregation of results should involve detailed yet authoritative standards (standard standards) in the process of measurement and aggregation.

I strongly support adding optional vocabulary and code attributes to the indicator element. The vocabulary codelist should be non-embedded to allow for its modification between upgrades. This is both an agile and a controlled procedure which should assuage Michael's fears concerning unilaterally defined libraries.

 Thanks, Bill. Agree that these elements should be optional, so the original title of this post is misleading. 

The intent of the indicator vocabulary codelist should be descriptive, not prescriptive. IATI is not in the business of passing judgment on the indicators that any given organization is using.

The point of IATI is to get information out to the world, not to take a political position on what constitutes good development practice and what doesn't. Different organizations take different approaches to measuring and consequently will have different kinds of indicators. 

Specifically to Michael's point, if you omit donor-defined libraries, you make it impossible for major donors to be able to report on what they're accomplishing. We may not like donors' indicators, or we might even think that they shouldn't "impose" indicators (although in that case I wonder how we then expect them to be able to tell us what happened with the money we entrusted them with). But it's not IATI's remit to tell people what to measure or how. These indicators exist in the real world, and if people can't use IATI to report on them, then they'll be forced to find some other way, which would be a real shame. 

This proposal has been discussed amongst the IATI Technical Team following the end of the initial suggestion phase of the v2.02 upgrade.

The IATI Technical Team supports the principal of this proposal as a non-mandatory addition to version 2.02 only.  Further work is needed to clarify the required changes to the schema and codelist status in order to implement this proposal, with further discussion encouraged in the meantime.

Talking about Results and aggregation can always be difficult. It looks like the main concerns with this have been addressed by making it optional, but some feedback from Bond members about the Results section came to me when having a read through:

- NGOs are nervous about using the Results section as it is a limiting way of describing their work. By describing their work in a particular way on a donor-led Standard, they are risking losing context and misrepresenting themselves to their funders and the public. We should be inviting feedback from them about the usefulness of the Results sections and the proposed indicators.

- I agree with Michael's point that vocabulary libraries should be sought and agreed upon through a collective process. For example, we could invite NGOs in a similar sectors to work together to define indicators. This would be a time-intensive exercise, but may be the only way to get valuable aggregated data.

- Through working with NGOs on IATI, I'm torn on whether having this as an option will have a benefit or not.

1) If the indicators are not relevant, they will not use them and there is unlikely to be increased use of the Results section. 

or

2) Sector Codes are not an unproblematic example of optional vocabulary. In order to appear in the sector aggregation on the visualisation website like d-portal.org, orgs MUST use the 5 Digit DAC. This may also be the case for DfID's DevTracker? These are public representations of data that orgs like to appear upon, which encourages them to shoehorn their work into sector codes that may not fit exactly. The peer pressure element of standardised indicators - even when optional - should not be dismissed. Future tools that work with Results data should take this into account and allow integration of 'private' vocabularies.

I agree that we should support 'private' vocabularies.

I also wholeheartedly agree that this shouldn't be about standardization, or about pressuring people to use donor-mandated indicators. We will never agree on a standard set of indicators that apply to everyone. Even small organizations have trouble harmonizing their indicators internally. I think the best we can hope for is eventually being able to cross-walk indicators from various sources - just to *observe* (without judgment) that indicator ABC measures the same thing as indicator 1.2.3.

Donor-mandated indicators are a fact of life for many development actors, for the simple reason that donors want to know what's being done with their money. So, again, I think we should not put IATI in the position of passing judgment on indicators.

<rant>

The international development community is facing a crisis of credibility because of its inability to account for what's being done with funds. Consider Haiti, where everyone from USAID to the Red Cross is facing fierce criticism for being unable to explain what they did with all the money that was put in their hands.

http://www.cgdev.org/blog/inspector-general-gives-usaid-failing-grade-haiti
http://www.npr.org/2015/06/03/411524156/in-search-of-the-red-cross-500-million-in-haiti-relief

So I find it baffling that we don't have a bigger sense of urgency about making results reporting possible using IATI. I'd frankly be very reluctant to give *my* money to an NGO that is "nervous" about being asked to quantify what they're accomplishing.

Certainly some things are easier to measure than others. But if we can't answer the simplest of questions like "how many houses did you build" or "how many households did you serve with clean water" or "how many children did you immunize" or "how many teachers did you train", the public is frankly justified in their skepticism that we're accomplishing anything at all.

It's all good and well to be transparent on how much money we're spending, but our transparency isn't very USEFUL to anyone in the real world if we're not also transparent about what we're achieving.

</rant>

We are currently supporting more humanitarian organisations to publish, and results in some ways are easier to quantify at least at output level: number of people helped, shelters built, child-friendly spaces established etc. I think the idea of identifying indicators to a particular vocabulary is very useful.

But I think Dan brings up a very relevant cautionary tale (the preference towards the OECD DAC Sector codes, and a practical example of how the IATI standard is not neutral. There is currently a bias towards mimicing DAC reporting, which is not helpful to other entities publishing to IATI. I'd hate to see that bias carried through in the results element, which is potentially one of the most useful elements of the whole Standard.

I would propose separating the bigger debate about a better results standard from the practical proposal here to at least be able to unambiguously refer to an indicator IF you are reporting quantitative results.

Making it possible to use your own indicator list, and (in another proposal) add basic disaggregation, will make the standard a lot more valuable to develop applications for quantitative results where appropriate.

Guidance on how this can be used is crucial.

There is a tendency of donors (well, DFID and DGIS) to just require everything, and of reference tools to favour the OECD DAC elements, which results in e.g. DAC sector code "shoehorning". But I think it would be a missed opportunity not to improve the technical basis with a few simple steps now. It also opens the way for organisations to use their own data in more meaningful ways.

We don't "require everything" Rolf - just a minimum standard ;) . But I take your point.

My apologies John, you're right, DFID isn't requiring "everything".

Please see below for the technical implementation that is suggested in the Version 2.02 - Formal Proposal:

New attributes: indicator/reference [MODIFIED 20-Nov-2015 due to an error with the original proposal]

• Addition of three new attributes to the activity standard: vocabulary, code and indicator-uri

• Example

• Existing

<indicator measure="1" ascending="1">

    ...

</indicator>

• Proposed

<indicator measure="1" ascending="1">

    ...

    <reference vocabulary="99" code="B1" indicator-uri="http://example.com/indicators.html" />

</indicator>

• Add definitions:
  reference: 'A standardised means of identifying the indicator from a code in a recognised vocabulary. Multiple vocabularies may be specified, but each vocabulary may be specified only once for each indicator.'
  [email protected]: ‘A code for a recognised vocabulary of indicators. The value for this field should appear in the IndicatorVocabulary codelist’
  [email protected]: ‘A code for an indicator defined in the specified vocabulary specified.’
  [email protected]: ‘If the vocabulary is 99 (reporting organisation), the URI where this internal vocabulary is defined’

New codelist: Indicator Vocabulary

• Addition of a new non-embedded codelist ‘IndicatorVocabulary’. Content of the list requires further consultation. Possible values include:

• 1: WHO Registry

• 2: Sphere Handbook

• 3: US Foreign Assistance Framework

• 4: World Bank World Development Indicators

• 5: UN Millennium Development Goals Indicators

• 6: UNOCHA Humanitarian Response Indicators

• 7: HIV/AIDS Indicator Registry

• 99: Reporting Organisation

• This codelist will be used for:

• result/indicator/reference/@vocabulary

I think the addition of the @indicator-uri attribute is a great idea. It should be clarified whether this is a link to an indicator vocabulary, or a link to an individual indicator. 

I'd advocate for making `reference` into an element, so that you can support multiple references. See this pull request https://github.com/HerbCaudill/IATI-Schemas/commit/ed0329cbdd47c01f8323c2713a1cab8eec335969 . The same indicator might be defined in multiple places (e.g. an implementing organization's framework, a donor's framework, and a host government framework). This is very, very common in practice. Allowing multiple references should go some of the way towards addressing concerns about donor-imposed frameworks. 

A couple of other thoughts:

1. There are lots of vocabularies out there. At the very least, I'd suggest using a larger number than 99 to refer to a private vocabulary. 
2. My preference would be to use alphanumeric codes for indicator vocabularies, because they're human-readable without a lookup (e.g. 'UN-MDG' instead of '5'). There's precedent for this in other codelists (e.g. Organization Registration Agency codes look like 'AF-CBR' and 'US-DOS').

Re @indicator-uri attribute, the indicator-uri links to a list, rather than an item, and this is consistent with other uses of uri attributes within elements that offer vocabulary specifications, and hopefully this is made clear in the attribute definitions:
“@vocabulary: ‘A code for a recognised vocabulary of indicators. The value for this field should appear in the IndicatorVocabulary codelist’
@code: ‘A code for an indicator from the list referenced in @vocabulary’
@indicator-uri: ‘If the vocabulary is 99 (reporting organisation), the URI where this internal vocabulary is defined’”

Re point 1, we think that the code ‘99’, now being uniform across other vocabularies and codelists used in the standard, should remain as a reserved code. This shouldn’t raise collision issues, as we are assigning these numeric codes, rather than them being derived from the codelist itself.

Re point 2, regarding the use of alphanumeric codes for indicator vocabularies, we’ve made a commitment in version 2.01 to move away from meaningful/human readable codes for several reasons. First is to move towards language neutrality where possible. Second is to make code management easier: there have been cases where third party codes have changed names and descriptions - the semantics of the codes hadn’t changed, but sometimes the names and descriptions undergo minor revisions. In this case it makes sense to divorce the reference from the meaning.

Finally, we should emphasise that organisation registration codes like ‘GB-COH’ and ‘NL-KVK’ are non-numerical because they are based on ISO codes (GB/NL), and because COH / KVK are region/language specific institutions (Companies House and Kamer van Koophandel respectively).

This proposal has been added as a GitHub issue, for inclusion in the version 2.02 development branch of the appropriate code repository: 

• New attributes: indicator/reference: https://github.com/IATI/IATI-Schemas/issues/275

• New codelist: Indicator Vocabulary: https://github.com/IATI/IATI-Codelists-NonEmbedded/issues/86

Herb, further to your post regarding making `reference` into an element, we have reviewed the proposal and we have realised that this was erroneously posted to this forum.

The above proposal has now been modified to add reference as an element, with attributes @vocabulary, @code & @indicator-uri.

This proposal has now been incorporated into a version 2.02 code repositories (see the above GitHub issue links for the schema and codelists for technical details). The relevant pages on a development version of the 2.02 reference and documentation site is available to view here:

• iati-activity/indicator/reference

• New codelist: Indicator Vocabulary

We welcome scrutiny on the implementation of this proposal and encourage the community to feedback and suggest solutions for any errors, omissions and problems. This should be done before Monday 7 December, when the process will commence to release version 2.02 as a live version of the IATI Standard. More information on the inspection phase is available here.