I am developing a WordPress plugin and wants to run a background task that calls an API and updates database table. Now, the API can only give results for 5 DB entries in one go and for 500 entries in my table, I have to make 100 API call. The API has allowed TPS quota of 1 and also in every 40 minutes, its old response expires which means I need to update my table if any entry is older than 40 minutes by making a new API call. And, all these DB entries have to be shown to the page viewer with latest data.
The solution that I came up with is scheduling a cron task that runs every minute and does API calls one after another for 25 seconds and then dies so that it doesn’t exceed PHP max execution time limit. And at time a customer comes, he has not to wait for API call or be throttled by API
But the problem is I can’t rely on wp-cron as it will be called only when page hit occurs(not like actual cron and since I am plugin developer I can’t schedule a system cron on my customer’s WordPress hosting environment). Also, if the person receives only one page hit in 3 hours, that first person in 3 hours has to either wait for those API calls to finish(not at all desirable or better to say feasible) or else he will not be shown the data.
Is there any other way to solve this problem of updating the DB entries via some background process that is neither slowing down the client and nor depending on page load. So even if 4 clients come in a day they all get latest data (data updated within last 40 minutes) as its already been updated in the background?
