Full history for the article here.

This is more of a work log than a blog, but I’m recording it as a post anyway.

Earlier I installed a firewall to make sure the server is safe. I tried getting X2Go running through a tunneled SSH connection, but the connection times out. Still have to look into this.

But

This week I downloaded nearly all of the files from DANS that conform to the following agreed specs:

• The data is of the ‘Archaeology’ type
• The data is in either text (nearly all csv) or .mid/.mif (MapInfo Interchange Format) type format
• The data is all open data, either as unrestricted (CC0) or ‘registered EASY’ users

The harvest was carried out in four batches of about 40,000 files each. Each batch took about 14 hours to complete.

The batches in total recorded only 23 files as erroneous: nine file downloads as unauthorized and fourteen reported an internal server error. I retried those 23 and was left out with only the nine unauthorized files.

In total, the download resulted in resulted in 156311 files with only nine omissions, which can be called successful.