radiocells-scanner-android
radiocells-scanner-android copied to clipboard
openbmap
CDN for some data
Hi again, long time no see :)
I'd like to put some money on a CDN so we can get DB updates a bit faster, do you think that's possible? How much traffic should we be handling if I set up the CDN?
@wish7code There is no interest on this? As said I have no problem on paying for the CDN myself but it'd be nice to be able to download the databases a bit faster.
I think to go in a strong future this is very important. Thanks for sharing your idea.
If it isn't too hard to install and it doesn't need more than lets say 5GB I could give some of my webspace. I rent a hosted server though I don't know if that's suitable. It is located in Switzerland and has unlimited traffic.
@klondi Let me really apologize for my late reply!! In fact it's such a generous offer and I'm really embarassed for not having answered before!
To get started we might distribute the raw measurement data (~1,8 GB) to different locations, secondly the wifi and cell catalogs (~1GB), i.e. the databases which are used for offline geolocation and within the Radiobeacon app. Does that sound reasonable? I am a bit unsecure, whether I listed all relevant files, because you also mentioned DB updates?
For the above mentioned files some webspace, we could sync automatically (SFTP, http upload, rsync), would be sufficient.
Currently our webspace is unmetered, so we never installed a traffic counter. I wouldn't expect excessive numbers of downloads, but we've got to keep in mind that each file is quite large..
Do you want to setup your very own CDN or get it hosted from a company like https://www.maxcdn.com/pricing/ ? We could add a donate button to the website as well...
Well for MaxCDN there is https://www.maxcdn.com/open-source/ . Also CloudFlare has a free plan which might be useful https://www.cloudflare.com/plans/
Generally I believe that plain hardware (+traffic) is available without extra cost as I guess many of us can access a lot of spare capacity. I guess the real shortage is free developer/admin time so anything which needs hand-holding/monitoring or development will be much harder to operate.
I guess the real shortage is free developer/admin time so anything which needs hand-holding/monitoring or development will be much harder to operate. True.
@wish7code Could you monitor traffic for a the next months? Like that we knew more about expected traffic.
Generally I believe that plain hardware (+traffic) is available without extra cost as I guess many of us can access a lot of spare capacity.
Thinks so too, we might start low profile by distributing the data file across multiple servers. All we would need in that stage would be some webspace. I guess in total we would stay below the 5GB threshold. A static ip would probably be a nice to-have, as it would ease setting up a load balancer.
Secondly, we would try to break the server in multiple independent components, e.g. geolocation server, map server, etc. This components could run independently on different server across the globe, while the core server would be resonsible for handling uploads and user authentication. Nevertheless this approach would probable need some sort of database replication.
Hi! @amilopowers my idea was just paying some CDN to take care of this.
Nowadays the ca 2 hours every database update requires make using openbmap in offline mode hard (to say the least).
@wish7code with most CDN providers what matters is the amount of data transfered over a month more than the size of the files (some actually provide hosting for the files aside from caching). If you have logs from the web accesses to fetch the databases I can help you parse them into something meaningful so we can get a higher bound on the use of traffic nowadays. I'd expect it to increase though as access to the openbmap data is made easier.
Finally the idea behind the database updates is allowing the clients to download a list of changes to apply to the database file instead of downloading the wholo database again, if these are kept for a reasonable amount of time (say for example daily for one month) then the update will be faster and more efficient than downloading the whole database again.
@klondi Sooorrry, being late again, my daytime job just kills me... For sure I could provide some logs. We don't have any trackers installed, so it basically would be the nginx webserver logs.
I was wondering wether it would make sense to arrange a skype session with everybody interested in the server architecture to discuss how we could further improve server architecture?
No worries man I also have a daytime job killing me :)
Nginx server should be enough I mostly care about the total size of the downloads per month so I can estimate how much will this cost.
As for the skype meeting no probs with it for me.