Is it allowed to scan your internet trafic and pick up logs

This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

  • Pretty much everything @Mon described is how I already described doing it. That is how EVE was doing it 12 years ago. Only difference is that instead of keeping it private to a guild I hope to open it up to anyone.

    Personally though I have never done anything like this, reading network packets and picking apart bytes. So taking me longer than I would like, but yay learning new things. Once I get that part figured out the backend stuff will be super easy.

    Appreciate @Mon sharing that info though, helps reaffirm everything I expected and that this plan should work.
    Twitter: https://twitter.com/regnerba
    Website: https://albion.regnerba.com

    The post was edited 2 times, last by FoxFour ().

  • I already did this on java, in the beta 1.
    After you isolate packets that "probably have data", assembly them is pretty easy.

    For this, i end with 4 "kind of packet".

    1 - packets with all the information, from the beggining of the json string, to the end (which usually have 1 or 2 items at most).
    2 - packets that have the Begining of the json string (When you list 3 or more items on the AH).
    3 - packets full of json string, but without the BOF and EOF (usually when you list more than 6 items, it will be the "second packet", and so on).
    4 - packets with the end of the json string (listing 6 items from the AH, it will be the 3rd packet or so).

    From that, you have to make a very nasty algorithm, which contemplate all cases.

    i did it, but was unable to contemplate when the packet with the EOF of the json string have less than 10 characters or so, because couldn't make a "matcheable" condition in that case.
    The best i could think of, was to "discard" a json that was not properly assembled.

    I ended just leaving the data on a jtable (no persistent) just to be able to read it (it was usefull to know who set the orders), but leave the project stand-by because the game was suffering a lot of changes.
    Maybe now that the game should not have big changes i will re-start it.

    Of course i found a lot of complications on the way, like repeat packets, missed packets, packets not matching, etc., but that was simply a matter of making the algorithm more "intelligent".

    Hope it help you.
    If i re-take the project, we can keep talking about it.
    Para comprar Albion Online con mi referido haz click aquí -->Comprar :)
    Gracias!!!

    The post was edited 1 time, last by Dragnark ().

  • Dragnark wrote:

    I already did this on java, in the beta 1.
    After you can isolate packets that "probably have data", assembly them is pretty easy.

    For this, i end with 4 "kind of packet".

    1 - packets with all the information, from the beggining of the json string, to the end (which usually have 1 or 2 items at most).
    2 - packets that have the Begining of the json string (When you list 3 or more items on the AH).
    3 - packets full of json string, but without the BOF and EOF (usually when you list more than 6 items, it will be the "second packet", and so on).
    4 - packets with the end of the json string (listing 6 items from the AH, it will be the 3rd packet or so).

    From that, you have to make a very nasty algorithm, which contemplate all cases.

    i did it, but was unable to contemplate when the packet with the EOF of the json string have less than 10 characters or so, because couldn't make a "matcheable" condition in that case.
    The best i could think of, was to "discard" a json that was not properly assembled.

    I ended just leaving the data on a jtable (no persistent) just to be able to read it (it was usefull to know who set the orders), but leave the project stand-by because the game was suffering a lot of changes.
    Maybe now that the game should not have big changes i will re-start it.

    Of course i found a lot of complications on the way, like repeat packets, missed packets, packets not matching, etc., but that was simply a matter of making the algorithm more "intelligent".

    Hope it help you.
    If i re-take the project, we can keep talking about it.
    Yea, I haven't found any sort of indication for indicating the end. However at the beginning it details how many entries are in the request. So I will just continue parsing until I have that many entries.
    Twitter: https://twitter.com/regnerba
    Website: https://albion.regnerba.com

  • FoxFour wrote:

    Dragnark wrote:

    I already did this on java, in the beta 1.
    After you can isolate packets that "probably have data", assembly them is pretty easy.

    For this, i end with 4 "kind of packet".

    1 - packets with all the information, from the beggining of the json string, to the end (which usually have 1 or 2 items at most).
    2 - packets that have the Begining of the json string (When you list 3 or more items on the AH).
    3 - packets full of json string, but without the BOF and EOF (usually when you list more than 6 items, it will be the "second packet", and so on).
    4 - packets with the end of the json string (listing 6 items from the AH, it will be the 3rd packet or so).

    From that, you have to make a very nasty algorithm, which contemplate all cases.

    i did it, but was unable to contemplate when the packet with the EOF of the json string have less than 10 characters or so, because couldn't make a "matcheable" condition in that case.
    The best i could think of, was to "discard" a json that was not properly assembled.

    I ended just leaving the data on a jtable (no persistent) just to be able to read it (it was usefull to know who set the orders), but leave the project stand-by because the game was suffering a lot of changes.
    Maybe now that the game should not have big changes i will re-start it.

    Of course i found a lot of complications on the way, like repeat packets, missed packets, packets not matching, etc., but that was simply a matter of making the algorithm more "intelligent".

    Hope it help you.
    If i re-take the project, we can keep talking about it.
    Yea, I haven't found any sort of indication for indicating the end. However at the beginning it details how many entries are in the request. So I will just continue parsing until I have that many entries.
    For the last packet, if you have at least one character of the key int the last "key:value", or the ":", you can match the value, but if in the last package you only have characters of the "value", then you are pretty much screw.
    If you find how to solve it, i will be pleased to hear you :P.

    edit: lol, didn't figure that about the number of entries. That is a very good data. It should be usefull!


    Redbowl wrote:

    what is all this nerd lingo? All I see are weird symbols.

    P.S. WTB button to click on and scan all items in AH for prices, kappa. 100$ US

    Make it easy to use

    imgur.com/a/6x5tf

    Please see above imgur, make it happen
    Actually, it doesn't work like that.

    The program will "capture" all the information that is listed on your screen (which have a limit of 100 or 200 items, i don't remember exactly).
    Anyway, the program capture the information automatically, but you have to search in-game action house manually.

    The idea is, if a lot of players use the program while playing the game and look for things on the auction house, and all that data is fetched on the same site, then that site should have a very update info of items on the auction houses.
    Para comprar Albion Online con mi referido haz click aquí -->Comprar :)
    Gracias!!!

  • Dragnark wrote:

    FoxFour wrote:

    Dragnark wrote:

    I already did this on java, in the beta 1.
    After you can isolate packets that "probably have data", assembly them is pretty easy.

    For this, i end with 4 "kind of packet".

    1 - packets with all the information, from the beggining of the json string, to the end (which usually have 1 or 2 items at most).
    2 - packets that have the Begining of the json string (When you list 3 or more items on the AH).
    3 - packets full of json string, but without the BOF and EOF (usually when you list more than 6 items, it will be the "second packet", and so on).
    4 - packets with the end of the json string (listing 6 items from the AH, it will be the 3rd packet or so).

    From that, you have to make a very nasty algorithm, which contemplate all cases.

    i did it, but was unable to contemplate when the packet with the EOF of the json string have less than 10 characters or so, because couldn't make a "matcheable" condition in that case.
    The best i could think of, was to "discard" a json that was not properly assembled.

    I ended just leaving the data on a jtable (no persistent) just to be able to read it (it was usefull to know who set the orders), but leave the project stand-by because the game was suffering a lot of changes.
    Maybe now that the game should not have big changes i will re-start it.

    Of course i found a lot of complications on the way, like repeat packets, missed packets, packets not matching, etc., but that was simply a matter of making the algorithm more "intelligent".

    Hope it help you.
    If i re-take the project, we can keep talking about it.
    Yea, I haven't found any sort of indication for indicating the end. However at the beginning it details how many entries are in the request. So I will just continue parsing until I have that many entries.
    For the last packet, if you have at least one character of the key int the last "key:value", or the ":", you can match the value, but if in the last package you only have characters of the "value", then you are pretty much screw.If you find how to solve it, i will be pleased to hear you :P.
    Speaking as someone who hasn't finished it yet and is still working on it, I was planning on looking for the closing "}" to know if I had the end of the JSON string. Will let you know how it goes. Hoping to spend time this evening and over the weekend working on it. If all goes well, HAHAHAHAHAHA it totally wont, I hope to have a prototype of the client ready by the end of the weekend.
    Twitter: https://twitter.com/regnerba
    Website: https://albion.regnerba.com

  • @FoxFour

    This is the snippet (i removed some useless pieces from it) to loop through the packets with market data, the code is a mess cause i didn't honestly care about how the client thing was.

    gist.github.com/ramonsaraiva/6…48f9fe5d6d480208e21490332

    This code is really old and they might've changed the packets structure, but i took a look and it seems like you can identify whenever it is a single or multiple packets with a sequence of bytes.

  • Mon wrote:

    @FoxFour

    This is the snippet (i removed some useless pieces from it) to loop through the packets with market data, the code is a mess cause i didn't honestly care about how the client thing was.

    gist.github.com/ramonsaraiva/6…48f9fe5d6d480208e21490332

    This code is really old and they might've changed the packets structure, but i took a look and it seems like you can identify whenever it is a single or multiple packets with a sequence of bytes.
    Thank you very much for sharing! Really do appreciate it. :)
    Twitter: https://twitter.com/regnerba
    Website: https://albion.regnerba.com

  • I just submitted a PR for making the network device configurable. Thanks for this!

    I would like to be able to send the data to a more configurable back-end without the need for a message queue. I think an API writing to a database would be sufficient and more simple. Having the client stay separate from ingestion will help. Having used Go professionally I think it is a great fit for the client, but I wouldn't want to build a non-critical web-app with it.

  • Aggrovate wrote:

    I just submitted a PR for making the network device configurable. Thanks for this!

    I would like to be able to send the data to a more configurable back-end without the need for a message queue. I think an API writing to a database would be sufficient and more simple. Having the client stay separate from ingestion will help. Having used Go professionally I think it is a great fit for the client, but I wouldn't want to build a non-critical web-app with it.
    Merged the PR, thank you!

    I am totally fine with configuring the backend the client API points to and just a good idea in general. Main reason I am building the backend in Go is just as an excuse to learn Golang. I haven't had the chance to use it in work, primarily deal in Python, so having a decent project to apply it to on the side is exciting to me. May I ask why you wouldn't use Go for the backend?

    If I was the one collecting the market data and maintaining a market web site for things then yea just stuffing it all in a database would work fine. The reason the backend is a bit more complicated is because it is meant to be an aggregation point and to relay the data to anyone who wants it. One of those people will probably be someone who stores it in a database and offers your standard market web site. However others might use it just to create price alerts, so they want access to that real time feed. Since this project only really works if a substantial number of people install the client I think it is really important for that real time feed to be shared with anyone who wants it. Otherwise everyone who wants the realtime feed is setting up their own instance and telling their guild/friends to use theirs.

    For that realtime feed I don't think putting the information into a database is the best first step. Others collecting from this service will want to put it in a database and thats great.

    One of the other parts that will probably be built later once/if the project takes off is a relay. Something that connects to the service I host and just repeats everything it gets out to anyone else listening to it. That would allow anyone to host a relay and share the burden hosting the network. Having the relays is the only reason the EVE version of this was able to stay active for so long and serve so many people.

    Currently playing around with the access to the feed being done over websockets. Pretty much every language has a decent websocket library and it allows me to host the backend far more easily in my Kubernetes cluster behind an HTTP loadbalancer.
    Twitter: https://twitter.com/regnerba
    Website: https://albion.regnerba.com

  • Bercilak wrote:

    * And we will also always check for other unfair advantages whatever they may be
    To be honest surprised SI didn't consider the market data being so easily accessible, and giving the OK to access it, an unfair advantage for those that could access it vs those that couldn't. Hopefully this AMDR project takes off and everyone can have good access to the data.
    Twitter: https://twitter.com/regnerba
    Website: https://albion.regnerba.com

  • What OP is describing is very shady.

    Basically building a database with the info posted in the OP you are creating a global AH that only you can view. You can see all the listings across the entire game as if each trader was linked together.

    I'm sure a few on here play Elder Scrolls Online. It has a decentralized auction house where nothing is linked together.

    That game does openly support add-ons and there is a particular add-on that essentially scans every store in the game and uploads the information to a website where you can search in real time what is listed.