Hi Richard Leeke,
I tried something new and I think I'm close.
I opened the shapefile in QGIS and using a QGIS plugin (MMQGIS) I converted the geometry of the shapefile to 'Linestrings'...
... then I used shapetotab, but I got a couple of warnings.
Using Tableau 8.2 I build the map...
...but it isn't precise, because they're segments of roads missing...
Do you've any thoughts about what's happening?
I have no idea what the errors mean either, however, based on some of the other issues I have been having with Tableau, I can safely assume something is mucked up on my computer…stupid thing…I swear I am going to take a hammer to it….
Business Analytics Supervisor
563.557.2504 ext. 6833
The Platinum Building l 137 Main l Dubuque, Iowa 52001
563.557.2504 | 866.225.0727 | fax 563.557.9180
<http://www.facebook.com/pages/Platinum-Supplemental-Insurance-Inc/155822301147218> <http://twitter.com/#!/pltnminsurance> <http://www.youtube.com/user/platinuminsurance/feed> <http://www.linkedin.com/company/platinum-supplemental-insurance-inc-?report%2Esuccess=se2U8YjzSfSLqmrvJWfkzmUYKl6uKkphatxlAbbto9OkvCyhhsMCP1nNWIEQv6pq2D50iVxJ>
I have done a quick bit of research on the error message, and all I can find is that the UTF-8 encoding isn't set up right on the PHP page...whatever the **** THAT means....
First thing is to work out why you get those warnings - you probably need to find those features in QGIS and see what they look like in the raw data.
Are the missing road segments visible in QGIS?
The missing road segments are visible in QGIS.
As you can see in image 1 (only motorways) the differences are small, but in countryside roads differences are enormous (image 2).
Hmmm. There seem to be more missing roads than the number of features that gave that warning message.
I'd still like to know what that warning is about. Can you have a look in QGIS and see if you can spot anything funny about the features that reported the warning (334, 441, etc)?
Also - the "original" pictures above from QGIS - were they from the original original file (the multipolyline or whatever it was called) or were they after you had converted it to something Shapetotab understands (or at least claims to understand)?
I'll try to take a look at your file at the weekend. I think you posted the original original. Could you also post your converted one? I'm aiming to do two things - work out where your missing roads have gone and see if I can support the original format. No promises on either count.
Hi Richard Leeke,
a) I found something curious. The attribute table of original shapefile (REDE_CALIBRADA_EP.shp) only've 898 rows, but the table of the converted geometry type to linestrings (linestrings.shp) have 1605 rows. As you can see in image below, some objectID in linestrings.shp are duplicate and exist objectID with different number of rows (e.g. A24) between the two tables.
b) They're from the converted geometry type to linestrings (linestrings.shp), but in QGIS linestrings.shp and REDE_CALIBRADA_EP.shp (ORIGINAL RAW) are equal visually.
c) ORIGINAL RAW -> Dropbox - Rede Calibrada.zip
LINESTRINGS CONVERTED -> Dropbox - LINESTRINGS.rar
I figured out what is wrong with your LINESTRING file. I don't know why it is like that - but I'm pretty sure you can fix it up pretty easily in QGIS, as I'll explain below.
First I'll just explain the reason for the different numbers of features in the original and converted files and the duplicate OBJECTIDs. That is simply because the original format was MULTILINESTRING - which means that each feature may have one or more line segments. When you converted it to LINESTRING, each of the line segments for a given feature in the original file becomes a new feature in the LINESTRING file - but they all have the OBJECTID of the original. So I'm just generating a new ID field based on the row count (in fact QGIS does the same internally).
Now to your missing roads. The issue is that a few of the features in the converted file appear to have stray newline characters inside the "roadnumber" field. In shapetotab, I'm exporting the shapes as a text file, with one feature per line in the file, and then parsing those lines and converting to CSV. The newlines cause some rows to be split across two lines, which I can't cope with.
So all you need to do is go into each of the rows causing problems (you can work out which ones from the warning messages I put out about invalid geometry.
Here's the first problem row. It's actually row 333 which is the problem - that gets written out to the text file as lines 333 and 334. The missing fields from line 333 aren't important for shapetotab, so it's at line 334 that I report the error.
Here's the row in QGIS - I've circled the extra character at the end of the roadnumber. You just need to delete that for all of the troublesome rows.
And here's how it comes out in the text file, split across two lines (I clipped most of the coordinates out).
I report a warning for lines which don't start with a valid geometry type (like "LINESTRING").
Just one extra trick to watch out for. As each corrupt feature is generating two rows in the text file, the line numbers I report as errors will be progressively further out of step with the QGIS numbering. I report these errors:
Warning, invalid geometry: '' for feature: 334 - generating dummy entry
Warning, invalid geometry: '' for feature: 441 - generating dummy entry
Warning, invalid geometry: '' for feature: 483 - generating dummy entry
Warning, invalid geometry: '' for feature: 711 - generating dummy entry
Warning, invalid geometry: '' for feature: 811 - generating dummy entry
Warning, invalid geometry: '' for feature: 852 - generating dummy entry
Warning, invalid geometry: '' for feature: 1271 - generating dummy entry
Warning, invalid geometry: '' for feature: 1333 - generating dummy entry
Warning, invalid geometry: '' for feature: 1399 - generating dummy entry
That means you will need to fix up rows:
Probably pays to check my maths - but I think those are the ones.
I'll also have a go and see if I can cope with MULTILINESTRING - should be easy I already do that for MULTIPOLYGON.
I've just added MULTILINESTRING and it seems to work OK. I'll just give it a bit more of a test before I post a new version. Should be tomorrow (it's nearly midnight here, so I'm calling it a night),
Not surprisingly those same 9 features cause the same problem with your original file - so instead of fixing up LINESTRING.shp you might want to fix the original. Here are the row numbers reported for the original file (you'll have to subtract 1, 2, 3 .. 8, 9 from the numbers given here as well).
Warning, invalid geometry: '' for feature: 178 - generating dummy entry
Warning, invalid geometry: '' for feature: 235 - generating dummy entry
Warning, invalid geometry: '' for feature: 262 - generating dummy entry
Warning, invalid geometry: '' for feature: 395 - generating dummy entry
Warning, invalid geometry: '' for feature: 443 - generating dummy entry
Warning, invalid geometry: '' for feature: 473 - generating dummy entry
Warning, invalid geometry: '' for feature: 699 - generating dummy entry
Warning, invalid geometry: '' for feature: 721 - generating dummy entry
Warning, invalid geometry: '' for feature: 762 - generating dummy entry
The MULTILINESTRING version seems to work OK except that a few hundred points come out with lat/lon coordinates of 0, 0. That seems to be something that is happening in the spatial libraries I'm using to convert the shapefile to lat/lon (WGS 84). Here's the map of all points:
And here after filtering out the zeros:
I have no idea why the GDAL libraries are setting all those points to (0, 0), but I tried converting the shapefile to lat/lon coordinates using QGIS (just load the shapefile, right click on it and choose save as, then set the CRS to WGS 84 - the lat / lon coordinate reference system that Tableau uses). Then I ran shapetotab on that.
That still gives one single point at (0, 0) which you need to filter out - but that's as close as I can get.
I'll post the new version of shapetotab on the Map Utility Downloads page in a minute.
I forgot to mention that I added a new column to the points file: line_segment_id. That is just a sequence number for the individual line segments within a MULTILINESTRING feature. That needs to be on Level of Detail as a dimension.
Hi ,Richard Leeke,
After so much time and headaches, finally we (you ) could do it.
And what small problem.
Here it's my result, after delete the stray newline characters and use the new shapetotab version.
Recently we upgraded at Tableau 8.2.2.
I used to work with your geohack tool, but now, after the upgrade I get an errror after trying to generate the shapes. Please find the error below
unsupported on-disk structure for file GEOCODING.FDB; found 11.2, support 11.
Can't call method "errstr" on an undefined value at tabgeohack.pl line 255.
So, I'm generating the csv file, I'm importing it in Tableau, close it, and when I try to run the --shapes command it gives me the above error.
Have you encountered this one before? Any idea how can I solve it?
Thanks a lot in advance
Yes, 8.2 broke it. I have published a new version which works for 8.2. There is a new thread in VizTalk called Map Utility Downloads which has links to the latest versions. Or you can scroll back a few pages in this thread and read all the discussion about it.
Just remembered that I never looked into your errors.
I've had a quick look and I'm not much the wiser than last time I replied. But I did find that the second error (the collations one) is very commonly reported by people using the Firebird database - and the whole reason for needing to upgrade tabgeohack for Tableau 8.2 is that Tableau has upgraded from version 2.1 to version 2.5 of the Firebird database for their geocoding data. The line where that error happens is in the code that deals with purging of unwanted geocoding data - which is completely unrelated to the line where the other error happens (where it is splitting up the latitude and longitude fields).
My only thought is that the issue appears to be in some way related to something about the particular configuration of your PC. Possibly conflicts with other software installed? Possibly some settings to do with character encoding?
The only thing I can suggest would be to try it on a completely different PC (say on your home PC rather than at work - just using another work one might not help if the issue is a conflict with something in your standard build).
And on your question about needing names as well as FIPS codes - to be honest I haven't done anything with FIPS codes - but I believe they are supported for geocoding. But I don't think there is any easy way to get Tableau to translate them into the names. I could be wrong on that though.