How Do I Get My Dog To Empty The Bowel

The potential of shared data as a service is already available

In EBICYS used to working with one of the most important assets of organizations: their data. For these are really good must be in a format that allows:

- Share them
- Debug
- Enrich
- Reusable
- We analyze
- Display

In many cases the format is not appropriate and only meet one of these objectives.

As an exercise we have taken data reconstruction grants awarded to the victims of the last major earthquake that struck Chile on 27/02/2010. The Ministry Housing published on its official website a list of more than 100,000 grants in a PDF file with a size of 75 MB which allows fairly (very large and slow to download) the first objective, namely to share, but this format only is useful for those who wish to audit one to one the list but fails to meet any of the other objectives mentioned missing the opportunity to get more out to them.

understand that the main objective was to communicate the list of beneficiaries but would be good places to start delivering state "data services", as do other states, for use and analysis. Using the parallel

with what happens in many organizations when faced with a draft Datamarts / Datawarehouse PDF file illustrates very well what happens in most cases:

- the data is not in the format we expected or were difficult to access
- need
data conversion - are not standardized (eg, cities Chillán, Chillan, Viña del Mar, Vina Del Mar, etc)
- do not meet all the same format and there is "dirt" in the same (lack of tabs / spaces)

just to mention a few things.

whole process of profiling, cleaning and normalization, extraction, transformation (the information enriching) is the one that took the longest amount of time for which data were useful.

As with any project of this type earlier stages were those who consumed 80-90% of the time and displays the remaining 10%. Finally

carry data at the level of detail and enriched in Google Fusion Tables (to be shared in a more easy and allows reuse) we display to allow their analysis and form a macro picture of the situation. Ultimately converted into information useful for analysis.

The tools we use are:

- FoxIt to convert PDF to text
- Google Refine, Excel 2010 for profiling, standardization and cleansing
- Google Fusion Tables as a data store and display

The following were the results.

WARNING: THESE DISPLAYS MAY CONTAIN ERRORS DUE TO THE FACTS WERE NOT AUDITED, THE SHORT TIME THAT HELD THE EXERCISE AND problems converting. This is done as an exercise to demonstrate the usefulness of SHARE DATA DISPLAYS AND PUBLIC ON THE NET. RECOMMEND ANY CONCLUSION FOR VALIDATING THE RESULTS FROM THE FILE IN PDF FORMAT POSTED ON THE SITE OF THE MINISTRY OF HOUSING.


The table with all the data can be browsed, queried y descargada   aquí



Los colores representan la concentración de subsidios siendo la escala : verde, amarillo, rojo y luego los íconos más grandes (mayor cantidad), amarillo y rojo.  Al hacer "click" en el ícono se muestra el total de subsidios de la comuna.