How to complicate site parsing?

Question

Parsing

There is an idea to create a website and mobile clients for it. Initially, it is planned to implement everything from a single API. That is, both the site and clients work with the same code that implements certain functionality.
There is a task to complicate API parsing and its unscrupulous use as much as possible. Since it is one thing to parse the site and html, and completely different - ready-made urls with json responses.
Serious guys use such a thing as a key to access the API. But in the same Google maps, it must be received by the developer, which does not suit us at all (it’s stupid to think that a user with an iPhone will receive some kind of key to access the application). And you will have to implement a mechanism for auto-obtaining a key, which completely kills the idea with keys.
The best option would be some kind of mechanism when both the site and the mobile client can trust some third party (Google on Android, Apple on iPhone). That is, the mobile client somehow connects with a third party, they communicate, and the third party has confirmation of its reality. The mobile client reports this information to us, we check it on a third party and admit it to the api. But, to be honest, I do not know such mechanisms.
The option of blocking by IP and other things is, of course, suitable, but this is the most obvious step and perhaps not the most effective one.

Answer 1

I think you are solving a problem that does not exist. If you have interesting, useful content that makes sense to parse, nothing will save you. As a last resort, they will write a proxy service that will parse html and issue json. Remember the fight between ICQ and Quip. The update came out a couple of hours after the protocol change.
If you are afraid that an android client will be made to your site, then it probably makes sense to just make your own right away? And if you are afraid of alternative clients, make yours better than the alternative. This is the only way.

Answer 2

For mobile clients:
1. After launching the application, collect information about the device and your client (hash-bodies, for example), encrypt, transfer to yourself.
2. On the server: write out an access key specifically for this device, generate a salt and save it in the database, return the key generation algorithm to the client with an admixture of salt and a timestamp.
3. The client each time an API request generates a signature based on salt, inf. about device and client (hash-bodies).
If the signature is correct and the response time was less than TIMEOUT, then API access is allowed.
For WEB (each time a page is called where there is interaction with the API):
1. Request a page from the server: save on the server: key1, time stamp, content of the page sent to the client
2. The client script from the page loads the ajax function for calculating key2 based on the entire page source using key1 and immediately returns the calculated value (key3) to the server.
3. On the server, check key3, and if the response time was less than TIMEOUT, then the server allows access to the API (API access is signed by key3).
All options, of course, bypass, but the task is much more complicated.
If anyone else has better options - I'll read it with pleasure.

Answer 3

Maybe you should try certificate-based authentication? Of course, it will not exclude the possibility of stealing a key from an android application, but it will complicate the task by an order of magnitude.

Answer 4

As another option. You can use antiDDOS protection. That is, if there are many requests from one IP, in a short period of time, to different urls / requests. It means that not an application, not a person looks at the site, but the parser pulls everything. You can safely ban.

Answer 5

Fully AJAX site. Like what happens when working with GWT. Since there is also a gwt-phonegap wrapper, you will also get a universal mobile application for the main platforms.

Answer 6

For a mobile application - encryption.
For the web version, also something binary as a client, also with encryption.
Flash, Java, Silverlight - worse (can be parsed). Chrome Native Client or a downloadable native binary plugin for working with the site is better.