The Albion Data Project - Beta Release

    • The Albion Data Project - Beta Release

      Sandbox Interactive on 22 Aug 17 wrote:

      Mandatory Disclaimer for 3rd party applications (added by Sandbox Interactive)
      • This application is a 3rd party project not endorsed or supported by Albion Online and Sandbox Interactive. It is however, as of August 22nd 2017, not explicitly forbidden by us. When using the app, you place your trust entirely into the hands of its developers.

      • The associated app, when installed on your computer, gets access to such computer and to traffic that it sends over the internet. This could theoretically be used with malicious intent and is a potential security risk.

      • When you us the app while playing the game, you bear the risk that it could be in breach of Albion Online's Terms and Conditions. If it is found to be in such breach, you personally bear responsibility for this. Sandbox Interactive has not audited the app in any way, hence cannot at this stage confirm or deny its compliance with the Terms and Conditions.


      We are super excited to announce the initial beta release of the Albion Data Project. The goal of this project is to gather and share publicly realtime information for Albion Online. This is achieved with a downloadable client that monitors network traffic specifically for Albion Online, identifies the relevant information, and then ships it off to a central server which distributes the information to anyone who wants it.

      The primary focus of this effort is to gather market data so that people can create tools to show the prices of items or estimate the cost of building items. Included in that is the current gold prices. Ideally we end up with something like EVE Central which was built off a tool very similar to this for EVE.

      What to know as a player
      At this moment we need people that are willing to help us beta test this. Testing requires downloading, installing, and running our client whenever you are playing Albion Online. There are not currently any major web sites that use this information, but before web sites and applications can use the information they need a supply of information. It's a bit of a chicken or egg kind of problem.

      This is a beta release however, so if you do take the time to help by installing and running this client please be patient and report and issues. There will be problems, things will break, but we will do our best to fix things as quickly as possible.

      If you would like to help test and begin providing some data download the client from this page: github.com/Regner/albiondata-client/releases

      What to know as a third-party developer
      On a technical level the distributed client is a Golang application watching network packets, collecting relevant information, and shipping it off to a NATS Server (nats.io/) hosted by us. As a developer you can connect to the NATS Server (nats://public:notsecure@ingest.albion-data.com:4222), subscribe to the relevant topics, and do with the messages as you please.

      Please note that this is meant to be a firehose of messages. If all you care about is the current-ish prices and want them via an API it is best to wait until someone develops a service, probably backed by this project, that offers the information via an HTTP API. This is meant to back services like that, for services that need the information as quickly as possible, and for services that need all of the information.

      A note on duplicate messages. As information comes into the NATS Server it is looked at and deduplicated over a 5 minute window. As a subscriber the goal is that you should only get the same message once every 5 minutes. This is of course open for change as we go however. The reason we are sending the same message at all is two fold.

      The simpler reason is that new people connecting to the network may have missed previous messages. Along with that however we don’t have a good way of noticing things like market orders completing. To remove market orders from your application the current best idea around is to keep track of the last time an order was seen, and then after not seeing it for X hours remove it as probably having been completed.

      Here are the current topics that as a subscriber you might care about:
      - goldprices.deduped
      - marketorders.deduped
      - mapdata.deduped

      We have also broken out the JSON structure for the messages we pass around into a separate Golang package. You can see them here (github.com/Regner/albiondata-client/tree/master/lib) as a reference for your own applications.

      While we will do our best not to introduce breaking changes into these message structures we do consider this a beta release and so things may indeed break.

      The future
      Right now we collect information on two things:
      1) Market orders, including their location
      2) Gold prices

      The Albion Online client receives information on many more things such as the plots at cities, ownership of zones for the world map, and much more. As we are able to identify this information, and get our services more stable, we will be adding more of those kinds of things to the list of what we look for and ship off. Market information and gold prices should give us a solid start though.
      If you have any other suggestions for things we should look at tracking please let us know. Preferably as a GitHub issue however here on the forums also works.

      Is this allowed
      This whole project was born out of someone asking Sandbox Interactive if they were allowed to look at the network traffic for Albion Online. Here is the response MadDave, Technical Lead at Sandbox Interactive for Albion Online, had to that thread:


      MadDave wrote:

      Our position is quite simple. As long as you just look and analyze we are ok with it. The moment you modify or manipulate something or somehow interfere with our services we will react (e.g. perma-ban, take legal action, whatever).
      All our client does is look at network packets and never modifies or manipulates them in any way which means our client falls well within what SBI says is OK.

      A huge thanks
      While I get to be the one that makes this post, this whole project would not have been possible without the help of a bunch of people working together. Here is a brief list of the people that have actively contributed to the project recently.

      @Fastdeath // @pcdummy86
      @FoxFour // @regnerba
      @gradiuscypher // @0xGradius
      @icras
      @Ultraporing // @Ultraporing
      Jeronica
      Nullsoldier
      haggen
      offner

      And of course Sanbox Interactive for making Albion Online and actually answering the question of if we are allowed to do this.
      I very well may have missed others as well and if i did I am deeply sorry.

      This is just the beginning
      We have some big plans for this project and are incredibly excited to see what others do with the data that comes from this project. If you’re interested in following along, learning more, contributing, or just chatting with us here are some links that may be of interest:

      albion-data.com
      This will probably become the hub of information for those looking to work with the data provided by the project.

      https://github.com/Regner/albiondata-client
      https://github.com/Regner/albiondata-deduper
      These are the two main repos that power the current project.

      discord.gg/8DDSjTs
      If you would like chat with us, or just hang out with others working on fansites and Albion related tools, join us in the above Discord server for Albion Online Fansites. This project mostly lives in the #proj-albiondata channel.

      Finally
      Thank you very much for taking the time to read through this post, a big thank you to those willing to install and test our client, and the biggest of thanks to everyone that has helped contribute to the project.
      Twitter: https://twitter.com/regnerba
      Website: https://albion.regnerba.com

      The post was edited 2 times, last by FoxFour ().

    • Hi @FoxFour

      this sounds like a cool project!

      For security reasons - as this is an installable app that gets quite a lot of access to the user's computer and web traffic - we would like to ask you to prominently display or link to the following disclaimer when discussing the app

      Sandbox Interactive on 22 Aug 17 wrote:

      Mandatory Disclaimer
      • This Albion Online data project is a 3rd party project not endorsed or supported by Albion Online and Sandbox Interactive. It is however, as of August 22nd 2017, not explicitly forbidden by us. When using the app, you place your trust entirely into the hands of its developers.
      • The associated app, when installed on your computer, gets access to such computer and to traffic that it sends over the internet. This could theoretically be used with malicious intent and is a potential security risk.
      • When you us the app while playing the game, you bear the risk that it could be in breach of Albion Online's Terms and Conditions. If it is found to be in such breach, you personally bear responsibility for this. Sandbox Interactive has not audited the app in any way, hence cannot at this stage confirm or deny its compliance with the Terms and Conditions.

      Within the app itself, we would like to ask you to make sure that it's users are made clearly aware that it is a 3rd party app. In particular, please do not use the Albion Online logo, Sandbox Interactive log, or any art assets from the game that could give the app an official appearance.

      Thanks!
    • Congratulations guys, this is looking awesome!

      Bogul wrote:

      As somebody with very little technical expertise, where do I find info on how to pull your data into my google docs sheet?

      Where do I start looking?

      Not possible directly I'm afraid. You'll need a proxy, something to consume the data, filter it (as it can contain duplicates) and then make it available in a format suitable for your sheets. But I hear people are already working on it, so just a matter of time. :)

      The post was edited 1 time, last by haggen ().

    • Korn wrote:

      Hi @FoxFour

      this sounds like a cool project!

      For security reasons - as this is an installable app that gets quite a lot of access to the user's computer and web traffic - we would like to ask you to prominently display or link to the following disclaimer when discussing the app

      Sandbox Interactive on 22 Aug 17 wrote:

      Mandatory Disclaimer
      • This Albion Online data project is a 3rd party project not endorsed or supported by Albion Online and Sandbox Interactive. It is however, as of August 22nd 2017, not explicitly forbidden by us. When using the app, you place your trust entirely into the hands of its developers.

      • The associated app, when installed on your computer, gets access to such computer and to traffic that it sends over the internet. This could theoretically be used with malicious intent and is a potential security risk.
      • When you us the app while playing the game, you bear the risk that it could be in breach of Albion Online's Terms and Conditions. If it is found to be in such breach, you personally bear responsibility for this. Sandbox Interactive has not audited the app in any way, hence cannot at this stage confirm or deny its compliance with the Terms and Conditions.

      Within the app itself, we would like to ask you to make sure that it's users are made clearly aware that it is a 3rd party app. In particular, please do not use the Albion Online logo, Sandbox Interactive log, or any art assets from the game that could give the app an official appearance.
      Thanks!
      Hey Korn,

      Thanks for the reply. I am just getting out of bed now but will do my best to find some time to get that information displayed on albion-data.com and as a message when the client starts up.
      Twitter: https://twitter.com/regnerba
      Website: https://albion.regnerba.com
    • SlayeS wrote:

      The uninstaller doesn't work. Can't uninstall :(
      Hmmmmm we will have to look into that. Not sure why it wouldn't work. Very sorry about that! If you would like to uninstall you should be able to just delete the files in program files for Albion Data Project. We have created an issue to track this and try and solve it as well: github.com/Regner/albiondata-client/issues/85
      Twitter: https://twitter.com/regnerba
      Website: https://albion.regnerba.com
    • I am very excited about this. I'll get to tinkering right away.

      While I have always looked poorly on people sharing secrets to wealth, sharing raw market data is entirely another matter. In sharing the data, you retain the need for market participants to come up with their own strategies. In sharing a market strategy, if it's actually an optimal one, you massively reduce its viability. And frankly most people touting "game breaking market strategies" haven't discovered anything optimal.

      So, thank you. This will help me discover more and more optimal market strategies.

      One note, (edit: some of) your links on this post to github are broken.

      The post was edited 3 times, last by zakaluka ().

    • For those of you with trust, security, or malware questions, let's take a look at this tool. (the developer won't get defensive about this - you shouldn't either - it must always be addressed)

      It's written in GOLANG, a compiled procedural language developed by google some years ago. GOLANG takes a lot of the memory sanity and garbage collection features from java's book and puts them into a more traditional compiler package similar to C++ (except with better data structuring, and frankly with a much cleaner environment)

      The project's source code is available directly from github, so you can read it and build yourself a local copy if you want. Now while most people won't do this usually, because it's a huge measure of paranoia for someone's personal computer, this should be taken as a gesture of good faith. The concept is called "open source". Look: my software doesn't do anything I'm reluctant to share with you.

      It uses a packet capture library called WINPCAP to retrieve the market data as you do your normal interactions with the various markets in Albion Online. He packages winpcap as a binary with his project, but winpcap itself is also open source. I don't know their business model, but the owner of winpcap looks like they're actually trying to make money with their tool somehow?

      All in all as a person familiar with these kinds of tools and the security risks that come along with them, I'm satisfied enough to install without doing a full read of the source at this point. But, if anyone is inclined to read the source, it's perfectly ok to ask more specific questions.

      Thumbs up from me.

      The post was edited 3 times, last by zakaluka ().

    • zakaluka wrote:

      For those of you with trust, security, or malware questions, let's take a look at this tool. (the developer won't get defensive about this - you shouldn't either - it must always be addressed)

      It's written in GOLANG, a compiled procedural language developed by google some years ago. GOLANG takes a lot of the memory sanity and garbage collection features from java's book and puts them into a more traditional compiler package similar to C++ (except with better data structuring, and frankly with a much cleaner environment)

      The project's source code is available directly from github, so you can read it and build yourself a local copy if you want. Now while most people won't do this usually, because it's a huge measure of paranoia for someone's personal computer, this should be taken as a gesture of good faith. The concept is called "open source". Look: my software doesn't do anything I'm reluctant to share with you.

      It uses a packet capture library called WINPCAP to retrieve the market data as you do your normal interactions with the various markets in Albion Online. He packages winpcap as a binary with his project, but winpcap itself is also open source. I don't know their business model, but the owner of winpcap looks like they're actually trying to make money with their tool somehow?

      All in all as a person familiar with these kinds of tools and the security risks that come along with them, I'm satisfied enough to install without doing a full read of the source at this point. But, if anyone is inclined to read the source, it's perfectly ok to ask more specific questions.

      Thumbs up from me.
      As you said, please check the project for your self's.
      Although I'm a contributor I highly encourage taking responsibility as end user and look into the software you install on your Pc.
      We are all very straight forward and are happy to answer any questions and doubts, even technical ones. We are all one community and should help each other.

      Cheers Ultraporing :D